Stats Accelerator versus multi-armed bandit optimization

Stats Accelerator's statistical model is a variation of multi-armed bandit. 

The intent of your test determines when you should an A/B test with Stats Accelerator enabled for the variation's traffic distribution, or run a multi-armed bandit optimization.

• To discover which variation improves your product with statistical certainty, run an experiment with Stats Accelerator. See When to use Stats Accelerator.
  • Helps you capture more value from your experiments by reducing the time to statistical significance.
  • It minimizes time by monitoring ongoing experiments and using machine learning to adjust traffic distribution among variations.
  • Traffic is routed to the variation most different from the baseline, regardless whether it is better or worse until statistical significance is reached; then traffic is routed to other variations.
• To optimize towards a primary metric, use Multi-armed bandit (MAB) optimizations. See When to use a multi-armed bandit optimization.
  • Helps you maximize the performance of your primary metric across your variations by dynamically re-allocating traffic to whichever variation is currently performing best.
  • It minimizes regret because traffic is routed to variations that show the best performance. The variation with the best performance has the highest revenue or the most conversions.
  • MABs help you extract value from the leading variation during the experiment lifecycle by avoiding the opportunity cost of showing sub-optimal experiences.

When to use Stats Accelerator

Stats Accelerator can cycle through many options quickly. Consider using Stats Accelerator whenever you want to run an A/B test that includes more than two variations against the baseline. 

Stats Accelerator shows when a variation is better than the baseline/original. Stats Accelerator answers "Which variation is the most unlike the baseline and also has the most optimistic performance projection?" 

The following examples show situations where Stats Accelerator are the best choice.

• Call To Action (CTA) – You can optimize a page for lead completion by testing many different copy options for a call to action.

• Landing page – Test several combinations of landing page copy, concept, and design to optimize for registrations, sign-ups, donations.

• Add-to-cart rate optimization – Drive an increase in your add-to-cart rate by showing visitors different default images on product pages or search results.  

• Search results optimization – Nudge users toward specific options (such as a travel site that encourages visitors to select a particular flight) by showing different results.

• Drive traffic to specific pages – By changing the location of the Recommended Content section, a media site can increase its clickthrough rates on recommended articles.

Set up an experiment with Stats Accelerator distribution mode 

Review the Stats Accelerator article to see how it works and how to interpret your results.

Create an experiment using Stats Accelerator in Web Experimentation

1. From the Experiments window, click Create New.

2. Select A/B Test from the drop-down menu.

3. Give your experiment a name, description, and URL to target, just as you would with any Optimizely experiment. Then click Create Experiment.

4. Create your variations in the Visual Editor. For experiments using Stats Accelerator, you need at least two variations and a baseline. So, three variations total. 

5. Click Metrics and choose your primary metric. Your experiment will use the primary metric to determine how traffic is distributed across variations.

6. Click Traffic Allocation. Under Variation Traffic Distribution, click the Distribution Mode drop-down list and select Stats Accelerator.

   

7. Test your experiment.

8. Click Start Experiment to launch your experiment.

Create an experiment using Stats Accelerator in Feature Experimentation

1. Follow the instructions on how to run an A/B test in the developer documentation.
2. Select Stats Accelerator for the flag rule's Distribution mode. For experiments using Stats Accelerator, you need at least two variations and a baseline. So, three variations total. 

   

When to use a multi-armed bandit optimization

A multi-armed bandit (MAB) maximizes conversions for short, temporary experiences. MAB answers "Which variation shows the largest reward?"

The following examples show situations where multi-armed bandit optimization is the better choice than a traditional A/B experiment.

• Promotions and offers – Users who sell consumer goods on their site often focus on driving higher conversion rates. You can offer special promotions that run for a limited time; for those, the changes you make are not intended to be permanent, and a MAB optimization sends more traffic to the over-performing variations and less traffic to the underperforming variations for the duration of the promotion. 

• Headline testing – Headlines are short-lived content that loses relevance over time. If a headline experiment takes as long to reach statistical significance as the lifespan of a headline, then any insights gained from the experiment are irrelevant in the future. Therefore, a MAB optimization lets you maximize your impact without worrying about balancing experiment runtime and the lifespan of a headline.

• Webinar – You can boost registration for webinars or other events by experimenting with several different versions of your landing page.

Set up a MAB optimization in Optimizely Experimentation

If you have not worked with MABs before, see Maximize lift with multi-armed bandit optimization.

Create an MAB optimization in Optimizely Web Experimentation

1. From the Experiments window, click Create New....

2. Select Multi-Armed Bandit.

3. Give your MAB a name, description, and a URL to target, just as you would with any Optimizely experiment. Then click Create Bandit.

4. Create at least two variations in the Visual Editor.

5. Click Metrics from the left-side navigation to choose your primary metric. Your MAB uses the primary metric to determine how traffic is distributed across variations.

   After you start your MAB, you cannot change the primary metric.

6. Test your MAB.

7. Click Start Multi-Armed Bandit to launch your optimization.

Create an MAB optimization in Optimizely Feature Experimentation

See Run a multi-armed bandit optimization to create an MAB flag rule in the Optimizely application and implement it in your code using the Feature Experimentation SDKs.

Related articles

• Maximize lift with multi-armed bandit optimizations

• Stats Accelerator