Calculate the statistical likelihoods of variations using the Stats Engine

When you run experiments, Optimizely Experimentation determines the statistical likelihood of each variation actually leading to more conversions on your metrics. Why does this matter? Because when you look at your results, you are probably less interested in seeing how a variation compared to the baseline and more interested in predicting whether a variation will be better than the baseline when implemented in the future. In other words, you want to ensure your experiment results pay off.

Optimizely Experimentation’s Stats Engine powers our statistical significance calculations. It uses a statistical framework that is optimized to enable experimenters to run experiments with high statistical rigor while making it easy for anyone to interpret results. Specifically, Stats Engine allows you to make business decisions on results as experiments are running, regardless of preset sample sizes and the number of goals and variations in an experiment.

As with all statistical calculations, it is impossible to predict a variation's lift with certainty. This is why the Results page displays Optimizely Experimentation’s level of confidence in the results that you see. This way, you can make sophisticated business decisions from your results without needing an expert level of statistical knowledge.

How Stats Engine works

To learn about the nuts and bolts of how Stats Engine works and how to interpret the results it generates, review these articles:

• Why Stats Engine results sometimes differ from classical statistics results

• Why Stats Engine controls for false discovery instead of false positives

• How and why statistical significance changes over time

Statistical concepts and techniques

You may also find it useful to read about the statistical concepts and techniques Stats Engine uses, which we describe in these articles:

• Statistical significance in Optimizely Experimentation

• Get to statistical significance faster with Stats Accelerator

• Confidence intervals and improvement intervals

• False discovery rate control

Further reading 

If you would like even more background on how Optimizely Experimentation's Stats Engine works, here are a few resources:

• Optimizely blog post, "Statistics for the internet age: The story behind Optimizely’s new stats engine" 

• Stats engine technical whitepaper

• Experimentation Insights on Optimizely.com