Bayesian statistics

Bayesian statistics provides a powerful framework for analyzing experiment data, offering a distinct approach compared to traditional Frequentist (Fixed Horizon) statistics. In Optimizely, this lets you incorporate prior knowledge (the current version of Bayesian statistics uses uninformed priors), gain intuitive insights, and make data-driven decisions with a focus on the probability of hypotheses.

Core concepts of Bayesian statistics

Bayesian statistics revolves around updating beliefs as new data becomes available. Key components include the following:

• Prior – Your probability assessment of the hypothesis or parameter before you observe the data, P(θ).
• Likelihood – The probability of the observed data under a given hypothesis or parameter value, P(x | θ), viewed as a function of θ.
• Posterior –  Your updated probability of the hypothesis or parameter after you observe the data, P(θ | x). Bayes' theorem produces the posterior by combining the prior and the likelihood.

Bayesian approaches

Optimizely uses the following Bayesian models for different metric types:

• Binary metrics (for example, Conversion Rate)
  • Assumes a Binomial likelihood for observed successes (conversions) and uses Beta priors for the success probabilities.
  • The posterior distribution for the success probabilities is also a Beta distribution.
  • For example, If you have a prior belief about a conversion rate, and then observe new conversion data, the Beta-Binomial model updates your belief to a posterior distribution. 
• Numeric metrics (for example, Average Order Value)
  • Assumes a Normal likelihood for observed data (sample mean) and uses Normal priors for the population mean.
  • The posterior distribution for the population mean is also a Normal distribution.
  • Data transformation – For non-normal data, the normal-normal model typically works fine due to the Central Limit Theorem, provided that the variance of the data is not close to infinity.
• Ratio metrics (for example, Revenue per Visitor) – The model generates the posterior sample by sampling from two independent posterior distributions: one for the numeric metric and one for the denominator metric.

Key Bayesian metrics in Optimizely results

When viewing your experiment results with Bayesian analysis enabled, you see the following metrics:

• Point estimate – The posterior mean of the relative improvement, representing the most likely effect size.
• Credible intervals – These intervals provide a range within which the true effect lies with a specified probability. For example, a 90% credible interval means the true value has a 90% chance of falling within that range.
• Chance to beat – This metric quantifies the probability that a treatment variant performs better than the control variant.
• Probability of being the best arm – For experiments with more than two variants, this indicates the probability that a specific variant is the top performer among all options.

Inference methods

Optimizely uses posterior distributions to perform inference:

• Sampling – Optimizely draws samples from the posterior distributions to estimate the metrics.
• Chance to beat calculation – The proportion of posterior samples where the treatment is better than control.
• Relative improvement – Optimizely calculates this as the expected value (mean) of the uplift based on the posterior samples.
• Credible intervals – They are estimated from posterior samples by taking the appropriate empirical quantiles (For example, the 5th and 95th percentiles).

By offering these Bayesian statistical methods, Optimizely empowers you to choose the analysis approach that best fits your experimental needs and provides the most actionable insights for your business.