Get to statistical significance faster with Stats Accelerator

Can I use my own algorithm?

You can programmatically adjust Traffic Allocation weights as needed using the REST API. Optimizely’s out-of-the-box Stats Accelerator feature was finely tuned based on millions of historical data and state-of-the-art work in bandits and adaptive sampling.

How much time will I save with Stats Accelerator?

When using Stats Accelerator, users typically achieve statistical significance two to three times faster than standard A/B/n testing. With the same amount of traffic, you can reach significance using two to three times as many variants simultaneously as possible with standard A/B/n testing.

How often does Stats Accelerator make a decision?

The model that dictates Stats Accelerator is updated hourly. Even for Optimizely Experimentation users with extremely high traffic, this is more than sufficient to get the maximum benefits of a dynamic, adaptive allocation. Please let us know if you require a greater or lower frequency of model updates, let us know.

What happens if I change the baseline on the Results page?

Selecting another baseline has no adverse impact, but the numbers may be challenging to interpret. We suggest keeping the original baseline when you analyze the Results data.

What happens if I change my primary metric?

The Stats Accelerator scheme reacts and adapts to the primary metric. If you change the primary metric mid-experiment, the Stats Accelerator scheme will change its policy to optimize that metric. For this reason, we suggest you do not modify the primary metric once you begin the experiment or campaign.

What happens when I pause or stop a variation?

We recommend that you refrain from doing this. Though Stats Accelerator will ignore those variations’ results data when adjusting traffic distribution amongst the remaining live variations, the variation that has been paused/stopped will exhibit conversion events due to delayed conversions.

Since Weighted Improvement is a weighted sum of the observed effect size within epochs, subsequent periods–after the variation’s been stopped–will yield a more significant effect size due to delayed conversion events but no decisions, resulting in a skewed Weighted Improvement that is misleading.

Suppose you believe that a variation is underperforming. In that case, we recommend letting Stats Accelerator determine this, after which it will minimize traffic to this variation (because it has reached statistical significance) so it can funnel the remaining traffic to the other variations. Otherwise, create a new A/B test with the variation removed.

How does Stats Accelerator handle revenue and numeric metrics?

For numeric metrics like revenue, the number of parameters to fully describe the distribution may be unbounded. Optimizely uses robust estimators for the first few moments (for example, the mean, variance and skew) to construct confidence bounds that are used, just like those of binary metrics.

How does Stats Accelerator work with Personalization?

In Optimizely Web Personalization, the Stats Accelerator option can be found in the settings for an individual experience. Stats Accelerator will automatically adjust traffic distribution between variations within campaign experiences. This will not affect the holdback. You should increase your holdback to a level typically representing uniform distribution to maximize benefit. For example, if you have three variations and a holdback, consider a 25% holdback.

What is the mathematical difference between Stats Accelerator and Multi-Armed Bandit?

In simple terms, if your goal is to learn whether any variations are better or worse than the baseline and take actions that have a longer-term impact on your business based on this information, use Stats Accelerator. On the other hand, if you want to maximize conversions among these variations, choose Multi-Armed Bandit. For a complete explanation, see our documentation on Multi-armed bandits vs. Stats Accelerator: when to use each.

In traditional A/B/n testing, a control schema is defined in contrast to several variants to be determined better or worse than the control. Typically, such an experiment is done on a fraction of web traffic to determine the potential benefit or detriment of using a particular variant instead of the control.

Suppose the absolute difference between a variant and control is significant. In that case, only a small number of impressions of this variant are necessary to confidently declare the variant as different (and by how much). On the other hand, when the difference is small, more impressions of the variant are necessary to spot this slight difference.

The goal of Stats Accelerator is to spot the significant differences quickly and divert more traffic to those variants that require more impressions to attain statistical significance. Although nothing can ever be said with 100% certainty in statistical testing, we guarantee that the false discovery rate (FDR) is controlled, which bounds the expected proportion of variants falsely claimed as having a statistically significant difference when there is no actual difference (users commonly specify to control the FDR at 5%).

In a nutshell, use Stats Accelerator when you have a control or default and investigate optional variants before committing to one and replacing the control. With Multi-Armed Bandit, the variants and control (if it exists) are on equal footing. Instead of reaching statistical significance on the hypotheses that each variant is either different or the same as the control, Multi-Armed Bandit attempts to adapt the allocation to the variant with the best performance.

Can you run a "blended experiment" where you start the experiment with Stats Engine and switch to Stats Accelerator mid-experiment?

No, do not combine A/B and Bandit methods in the same experiment. Do not change your audience distribution model mid-experiment. Combining A/B and Bandit methods ad-hoc will introduce severe bias and destroy the reliability and interpretability of your results for business decisions. 

For example, suppose Stats Accelerator (or any other bandit model) starts tunneling traffic to branch A. There would be a disproportionate number of conversions on branch B because some of the conversions are from people assigned in the test's initial A/B stage (i.e., Stats Engine). This bias would tamper with the validity and interpretability of the results.

Additionally, the choice to ad-hoc splice in different audience models in the same experiment may lead to an example of the peeking problem. Essentially, the experimenter would manually and arbitrarily interrogate the experiment midway through, vastly inflating the chance of a false positive result. A false positive result occurs when a conclusive result is reported between two variations when there is no underlying behavior difference between them. 

How does Stats Accelerator handle conversion rates that change over time and Simpson's Paradox?

Time variation depends on the underlying distribution of the metric value on time. Time variation occurs when a metric’s conversion rate changes over time. Stats Engine assumes this distribution is identically distributed.

Time variation is caused by changes in the underlying conditions that affect visitor behavior. Examples include more purchasing visitors on weekends; an aggressive new discount that yields more customer purchases; or a marketing campaign in a new market that brings in many visitors with different interaction behavior than existing visitors.

Optimizely Experimentation assumes identically distributed data because this assumption enables continuous monitoring and faster learning (see the Stats Engine article for details). However, Stats Engine has a built-in mechanism to detect violations of this assumption. When a violation is detected, Stats Engine updates the statistical significance calculations. This is called a “stats reset.”

Time variation affects experiments using Stats Accelerator because the algorithms adjust the percentage of traffic exposed to each variation during the experiment. This can introduce bias in the estimated improvement, known as Simpson's Paradox. The result is that stats resets may be much more likely to occur. (See the weighted improvement section for more information.)

The solution is to change the way the improvement number is calculated. Specifically, Optimizely Experimentation compares the baseline conversion rates and variation(s) within each interval between traffic allocation changes. Then, Optimizely computes statistics using weighted averages across these time intervals. For example, the difference in observed conversion rates is scaled by the number of visitors in each interval to estimate the true difference in conversion rates. This estimate is represented as a weighted improvement.

Why is Stats accelerator routing traffic away from well-performing variations/towards poorly performing variations?

Stats accelerator is looking to minimize time, not regret. That means it routes traffic towards branches that show the most distinct from the baseline, regardless of whether that performance is positive or negative.

This allows stats accelerator to achieve statistical significance on more branches more quickly by routing traffic to where insight is promising but statistical significance has not yet been reached.