- Optimizely Web Experimentation
- Optimizely Performance Edge
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
If you have been running A/B tests, you have probably wondered: why my experiment not reaching statistical significance?
Statistical significance is a measure of how unusual your experiment results would be if there was actually no difference between your variation and baseline and the difference in lift was due to random chance alone. In other words, it is a good indicator of how well the results of the sample you tested will reflect reality. On the journey of experience optimization, your speed of travel is tied to your success in getting statistically significant results.
Fortunately, savvy experiment design and an understanding of how statistical significance works under the hood will help you reach conclusive results.
This article provides a few tips on reaching statistical significance. We also touch on related concepts in other articles, including How long to run an experiment and Use minimum detectable effect to prioritize experiments. Even though we repeat some of the same principles here, we recommend that you read those as well.
Changes are too small
Sometimes, a small change can make a huge difference. Other times, modest adjustments do not make big enough waves to push your experiment to statistical significance.
If your revision is minor, its impact on your baseline conversion rate is likely to be small too. Stats Engine picks up this small difference but takes longer to decide whether it is a chance fluctuation or a lasting change in visitor behavior.
Check out the chart below to see how smaller improvements over the baseline require larger sample sizes (and time) to declare a statistically significance result.
Best practice
When you design an experiment, consider making changes that will significantly impact your visitor's experience–whether the change itself is big or small.
Low baseline
The most important metrics to a business sometimes have relatively low baseline conversion rates. In ecommerce, for example, the "purchase" conversion rate is a relatively low-frequency event: often below 3%.
Low baseline conversion rates affect the time it takes to reach statistical significance. In the chart above, note the difference in traffic required to reach significance for a 1% versus a 5% baseline.
Best practice
While it is important to track how experiments affect key metrics, it is only sometimes possible to directly capture the impact of that infrequent event in a timely manner. When this is the case, use a metric with a higher baseline to stand in for the other and measure success.
Imagine that you are optimizing the homepage of an ecommerce site with a banner that prompts visitors to visit the electronics category. You expect more visitors to click the banner, view electronics, and purchase. But the baseline conversion rate for purchases is relatively low. You have a limited amount of time to run this experiment; it will take too long to reach statistical significance.
Instead of measuring success in purchases, you set your primary metric to track clicks to the banner. That way, you do not have to wait for significance to travel down the funnel to decide whether the variation wins or loses. You measure the impact of your experiment directly in clicks, where you made the change. And, you can extrapolate that win to estimate your experiment's impact on revenue.
Too many goals
Stats Engine makes a distinction between primary and secondary metrics. The more secondary (or monitoring) metrics you add to an experiment, the longer it may take to reach statistical significance.
Best practices
Be strategic when deciding what metrics to track in an experiment. Add all the critical goals, even if it is ten or more. But do not track goals that are not crucial in deciding whether an experiment is a success or failure for your business needs.
For example, you are optimizing the search bar on your homepage. You are tempted to track how your changes impact clicks on your customer support widget. While customer support is a valid consideration, it may not be crucial to measure for this particular experiment.
Return to your hypothesis. Does the impact on support tell you whether the hypothesis is valid or not? If not, this goal may just get in the way of reaching statistical significance. Avoid adding it to this particular experiment.
<%= heading %>
