Experimentation distribution modes

Stats Accelerator's statistical model is a variation of multi-armed bandit. 

The intent of your test determines when you should run an optimization using Stats Accelerator, multi-armed bandit (MAB), or contextual multi-armed bandit (CMAB) distribution modes for the variations' traffic allocation.

• To discover which variation improves your product with statistical certainty – Run an experiment with Stats Accelerator. See the When to use Stats Accelerator section.
  • Helps you capture more value from your experiments by reducing the time to statistical significance.
  • It minimizes time by monitoring ongoing experiments and using machine learning to adjust traffic distribution among variations.
  • Traffic is routed to the variation most different from the baseline, regardless of whether it is better or worse until statistical significance is reached; then, traffic is routed to the other variations.
• To optimize towards a primary metric – Use multi-armed bandit (MAB) optimization. See When to use a MAB optimization.
  • Helps you maximize the performance of your primary metric across your variations by dynamically re-allocating traffic to whichever variation is currently performing best.
  • It minimizes regret because traffic is routed to variations with the best performance. The variation with the best performance is the one that has made the most impact on the primary metric.
  • MABs help you extract value from the leading variation during the experiment lifecycle by avoiding the opportunity cost of showing sub-optimal experiences.
• To optimize towards a primary metric and user attributes – Use a contextual multi-armed bandit (CMAB) (beta). See When to use a CMAB.
  • Helps you deliver the most personalized variation to each user by dynamically re-allocating traffic based on the primary metric and user attributes.
  • It maximizes impact because each user is served the most personalized variation depending on user context. For every new session, a user will receive the winning variation to increase conversion rates.
  • CMABs are ideal to be run on pages where the content is not changed too frequently, such as the homepage, as model performance improves over time.

When to use Stats Accelerator

Stats Accelerator manipulates traffic to minimize time to statistical significance. It monitors ongoing experiments and uses machine learning to adjust traffic distribution among variations.

The following examples show situations where an A/B test with Stats Accelerator is the best choice.

• Call To Action (CTA) – Optimize a page for lead completion by testing many different copy options for a call to action.

• Landing page – Test several combinations of landing page copy, concept, and design to optimize for registrations, sign-ups, and donations.

• Add-to-cart rate optimization – Increase your add-to-cart rate by showing visitors different default images on product pages or search results.  

• Search results optimization – Nudge users toward specific options (such as a travel site encouraging visitors to select a particular flight) by showing different results.

• Drive traffic to specific pages – Increase clickthrough rates on recommended articles by changing the location of the Recommended Content section.

See Stats Accelerator for information.

When to use MABs

MABs maximize conversions for short, temporary experiences. MABs answer, "Which variation shows the largest reward?"

The following examples show situations where a MAB optimization is better than a traditional A/B experiment.

• Promotions and offers – Users who sell consumer goods on their site often focus on driving higher conversion rates. You can offer special promotions that run for a limited time; for those, the changes you make are not intended to be permanent, and an MAB optimization sends more traffic to the over-performing variations and less traffic to the underperforming variations for the duration of the promotion. 

• Headline testing – Headlines are short-lived content that lose relevance over time. If a headline experiment takes as long to reach statistical significance as the lifespan of a headline, then any insights gained from the experiment are irrelevant in the future. A MAB optimization lets you maximize your impact without worrying about balancing experiment runtime and the lifespan of a headline.

• Webinar – You can boost registration for webinars or other events by experimenting with several different versions of your landing page.

When to use CMABs (beta)

This feature is currently in beta. Contact your Customer Success Manager to learn more.

A contextual multi-armed bandit (CMAB) maximizes conversions by showing variations that match user attributes. CMABs score variations based on impact on the primary metric and user attributes to serve the most personalized variation to every user.

You should use CMABs when user characteristics (such as device, location, or interests) impact what variation they would prefer to see. For example, a financial institution collects user attributes on credit cards, credit scores, loans, annual income, and spending categories. They build variations for their current offers of a travel credit card, auto loan refinancing, and a business credit card.

For a standard MAB, you would choose a primary metric, such as a call-to-action click, and display random variations to users. When one variation has a greater impact on the primary metric, more visitors are exposed to that variation. This is useful to capitalize on conversions during a short cycle like a weekend sale.

With a CMAB, you can personalize the user experience and ensure the variation chosen best fits the visitor's context. If the visitor has spent much money on air travel in the last six months, they would likely get the travel credit card offer. A user with a high-interest car loan likely sees the auto loan refinance offer. A user who recently opened a small business likely sees the business card offer.

Other use cases for CMABs include the following:

• Personalized offers – Personalize the offers users see based on their attributes, such as products owned or viewed, to increase conversion rates.
• Members-only sale – Deliver variations based on the user's membership for a members-only sale or something similar. 
• Time-based content delivery – Increase user engagement during specific times of day by showing different types of content or offers, such as morning promotions and evening discounts.
• Location-based offers – Target a user's location with region-specific promotions or discounts to increase conversion rates in different regions.
• Behavior-triggered messaging – Show different messages or call-to-actions based on recent behavior, such as reminders to complete a purchase or suggestions for related products, to reduce cart abandonment rates and increase completed purchases.
• Dynamic pricing strategies – Use different pricing strategies or discounts based on a user's browsing history, purchase intent, or location to optimize pricing and maximize revenue while remaining competitive.