Bayesian statistics

Bayesian statistics provides a powerful framework for analyzing experiment data, offering a distinct approach compared to traditional Frequentist (Fixed Horizon) statistics. In Optimizely, this lets you incorporate prior knowledge (the current version of Bayesian statistics uses uninformed priors), gain intuitive insights, and make data-driven decisions with a focus on the probability of hypotheses.

Core concepts of Bayesian statistics

Bayesian statistics revolves around updating beliefs as new data becomes available. Key components include the following:

• Likelihood – The probability of observing your experiment data given a specific hypothesis is true.
• Posterior probability – The updated probability of your hypothesis being true after considering the observed data. Optimizely calculates this using Bayes' Theorem.

Optimizely's Bayesian approaches

This approach uses established Bayesian models for different metric types:

• Binary metrics (for example, Conversion Rate)
  • Assumes a Binomial likelihood for observed successes (conversions) and uses Beta priors for the success probabilities.
  • The posterior distribution for the success probabilities is also a Beta distribution.
  • Example: If you have a prior belief about a conversion rate, and then observe new conversion data, the Beta-Binomial model updates your belief to a posterior distribution. 
• Numeric metrics (for example, Average Order Value)
  • Assumes a Normal likelihood for observed data (sample mean) and uses Normal priors for the population mean.
  • The posterior distribution for the population mean is also a Normal distribution.
  • Data transformation – For non-normal data, the normal-normal model typically works fine due to the Central Limit Theorem, provided that the variance of the data is not close to infinity. In addition, Optimizely can apply techniques like log transformation or inverse hyperbolic sine transformation to make the data more amenable to normal distribution assumptions.
• Ratio metrics (for example, Revenue per Visitor)
  • Combines the models for binary and numeric metrics, often treating the numerator and denominator separately and then combining their posterior samples.

Key Bayesian metrics in Optimizely results

When viewing your experiment results with Bayesian analysis enabled, you see the following metrics:

• Point estimate – This is the posterior mean of the relative or absolute improvement, representing the most likely effect size.
• Credible intervals – These intervals provide a range within which the true effect lies with a specified probability. For example, a 90% credible interval means there is a 90% chance the true value is within that range.
• Chance to beat – This metric quantifies the probability that a treatment variant performs better than the control variant.
• Probability of being the best arm – For experiments with more than two variants, this indicates the probability that a specific variant is the top performer among all options.

Inference methods

Optimizely uses the following posterior distributions to perform inference:

• Sampling – Optimizely draws random samples from the posterior distributions to estimate the metrics.
• Chance to beat calculation – The proportion of posterior samples where the treatment's effect is greater than the control's effect.
• Absolute and relative improvement – Optimizely calculates this as the expected value (mean) of the posterior distribution of the difference or ratio between treatment and control effects.
• Percentile-based credible intervals – Optimizely derives these directly from the ordered posterior samples.

By offering these Bayesian statistical methods, Optimizely empowers you to choose the analysis approach that best fits your experimental needs and provides the most actionable insights for your business.