Stats Accelerator compared to multi-armed bandit optimizations

Stats Accelerator's statistical model is a variation of multi-armed bandit. 

The intent of your test determines when you should an A/B test with Stats Accelerator enabled for the variation's traffic distribution or run a multi-armed bandit optimization.

• To discover which variation improves your product with statistical certainty – Run an experiment with Stats Accelerator. See the When to use Stats Accelerator section.
  • Helps you capture more value from your experiments by reducing the time to statistical significance.
  • It minimizes time by monitoring ongoing experiments and using machine learning to adjust traffic distribution among variations.
  • Traffic is routed to the variation most different from the baseline, regardless of whether it is better or worse until statistical significance is reached; then, traffic is routed to the other variations.
• To optimize towards a primary metric – Use a Multi-armed bandit (MAB) optimization. See the When to use a multi-armed bandit optimization section.
  • Helps you maximize the performance of your primary metric across your variations by dynamically re-allocating traffic to whichever variation is currently performing best.
  • It minimizes regret because traffic is routed to variations that show the best performance. The variation with the best performance has the highest revenue or the most conversions.
  • MABs help you extract value from the leading variation during the experiment lifecycle by avoiding the opportunity cost of showing sub-optimal experiences.

When to use Stats Accelerator

Stats Accelerator manipulates traffic to minimize time to statistical significance. It monitors ongoing experiments and uses machine learning to adjust traffic distribution among variations.

The following examples show situations where an A/B test with Stats Accelerator is the best choice.

• Call To Action (CTA) – You can optimize a page for lead completion by testing many different copy options for a call to action.

• Landing page – Test several combinations of landing page copy, concept, and design to optimize for registrations, sign-ups, and donations.

• Add-to-cart rate optimization – Drive an increase in your add-to-cart rate by showing visitors different default images on product pages or search results.  

• Search results optimization – Nudge users toward specific options (such as a travel site that encourages visitors to select a particular flight) by showing different results.

• Drive traffic to specific pages – By changing the location of the Recommended Content section, a media site can increase its clickthrough rates on recommended articles.

When to use a multi-armed bandit optimization

A multi-armed bandit (MAB) maximizes conversions for short, temporary experiences. MAB answers, "Which variation shows the largest reward?"

The following examples show situations where multi-armed bandit optimization is better than a traditional A/B experiment.

• Promotions and offers – Users who sell consumer goods on their site often focus on driving higher conversion rates. You can offer special promotions that run for a limited time; for those, the changes you make are not intended to be permanent, and an MAB optimization sends more traffic to the over-performing variations and less traffic to the underperforming variations for the duration of the promotion. 

• Headline testing – Headlines are short-lived content that loses relevance over time. If a headline experiment takes as long to reach statistical significance as the lifespan of a headline, then any insights gained from the experiment are irrelevant in the future. An MAB optimization lets you maximize your impact without worrying about balancing experiment runtime and the lifespan of a headline.

• Webinar – You can boost registration for webinars or other events by experimenting with several different versions of your landing page.

Mathematical difference between Stats Accelerator and a multi-armed bandit

In traditional A/B/n testing, a control schema is defined in contrast to several variants to be determined better or worse than the control. Typically, such an experiment is done on a fraction of web traffic to determine the potential benefit or detriment of using a particular variant instead of the control.

Suppose the absolute difference between a variant and control is significant. In that case, only a small number of impressions of this variant are necessary to confidently declare the variant as different (and by how much). On the other hand, when the difference is small, more impressions of the variant are necessary to spot this slight difference.

The goal of Stats Accelerator is to spot the significant differences quickly and divert more traffic to those variants that require more impressions to attain statistical significance. Although nothing can ever be said with 100% certainty in statistical testing, Optimizely guarantees that the false discovery rate (FDR) is controlled, which bounds the expected proportion of variants falsely claimed as having a statistically significant difference when there is no actual difference (users commonly specify to control the FDR at 5%).

In a nutshell, use Stats Accelerator when you have a control or default and investigate optional variants before committing to one and replacing the control. With Multi-Armed Bandit, the variants and control (if they exist) are on equal footing. Instead of reaching statistical significance on the hypotheses that each variant is different or the same as the control, Multi-Armed Bandit attempts to adapt the allocation to the variant with the best performance.

Related articles

• Maximize lift with multi-armed bandit optimizations

• Stats Accelerator

• History of Stats Accelerator and multi-armed bandits