You are using Cloud Run to host a global ecommerce web application. Your company’s design team is creating a new color scheme for the web app. You have been tasked with determining whether the new color scheme will increase sales. You want to conduct testing on live production traffic. How should you design the study?
To determine whether the new color scheme will increase sales, you should employ A/B testing by using an external HTTP(S) load balancer to route a predetermined percentage of traffic to two different color schemes of your application. This method allows you to directly compare user interactions and sales data between the existing color scheme and the new one, ensuring a statistically significant difference can be identified if present. This setup provides a controlled way to evaluate the impact of the color scheme change on live production traffic.
Correct answer is A. This is classic A/B testing. Since you already have a new version, built into an image, all you need do is to use the load balancer to split traffic going to old version and new version. See: https://cloud.google.com/load-balancing/docs/l7-internal/traffic-management#traffic_actions_weight-based_traffic_splitting. Note that global load balancers can route to serverless services. https://cloud.google.com/load-balancing/docs/https/setting-up-https-serverless
This is the best way to test the new color scheme on live production traffic. By enabling a feature flag, you can display the new color scheme to a subset of users while keeping the old color scheme for the rest of the users. This will allow you to compare sales between the two groups of users and determine whether the new color scheme has a statistically significant impact on sales.
A is right. D is specifying 50% of the users which is not correct. In really the traffic split is 80-20 or 75-25 ratio. This is a specialized version of canary depolyments.
A is correct.
Considering the importance of traffic analysis and the need for precise control over traffic distribution for a global ecommerce web application, Option A is likely the better choice. This option allows for detailed monitoring and analysis of user interactions with different color schemes, offering clear insights into which version performs better in terms of sales. The use of an external HTTP(S) load balancer for traffic routing provides a more controlled environment for conducting such a study.
Besides Cloud Run alone doesn't provide Feature Flag solution that requires Firebase Remote Config feature. For an A/B test you have to fix what users are using what version. Option A doesn't not provide a complete solution for something that is crucial. So, besides both seems incomplete, if that request was made to me, I'd implement as a Feature Flag using Firebase Remote Config