Change the statistical significance setting in Optimizely Experimentation

  • Updated

This topic describes how to:
  • Decide whether to change the statistical significance settings for your projects
  • Find and adjust the statistical significance setting for your projects

When you run experiments, Optimizely Experimentation's Stats Engine constantly analyzes your results to determine whether your results are statistically significant: how confident you can be that the results actually reflect a change in your visitors' behavior, not just noise or randomness. With the statistical significance setting, you can set a project-wide significance level at which you would like Optimizely Experimentation to declare significant results (winners and losers) on the Results page.

Trade-offs of changing the statistical significance setting

You should be aware of certain trade-offs associated with changing the statistical significance setting. In general, a higher significance setting is more accurate and increases the time required for Optimizely Experimentation to declare significant results because it requires a larger sample size. A lower statistical significance level decreases the amount of time needed to declare significant results, but lowering the statistical significance setting also increases the chance that some of the results will be false positives.

Changing your statistical significance setting will instantly affect all currently running experiments. If your experiment has a goal with an 85% statistically significant winner, and you change your statistical significance setting from 90% to 80%, the next time you load your Results page, you will see a winner (85% > 80%). Your difference intervals will also shrink to reflect the reduced need for confidence.

Change the statistical significance setting

The statistical significance setting is listed under Settings > Advanced. After you adjust the setting, scroll down and click Save. Allow a few minutes for the new setting to take effect.

In the Optimizely Experimentation UI:

stat-sig-setting.png

Summary:

  • In the Results Page, the primary goal statistical significance is calculated independently from all other goals. 
  • The primary goal is not impacted by the secondary goals and secondary goals are not impacted by primary goals.
    • If you add many goals and variations that have tiny to no improvement, then it will take longer to reach significance on all your secondary goals.
    • If you add goals and variations that have large improvements, it makes it easier (faster) to find winners and losers overall.
    • If all your goals and variations have roughly the same improvement, it will take roughly the same time to find winners and losers no matter how large the group.
    • This removes the hidden risks associated with testing more goals and variations because no matter how many you test, you will now see a correctly adjusted statistical significance.
  • When swapping out the primary goal, this by itself actually does not trigger a recalculation of the results on this goal as the new primary.