Configure a Frequentist (Fixed Horizon) A/B test

Follow these steps to configure a Frequentist (Fixed Horizon) test in Optimizely Feature Experimentation. This configuration lets you run an A/B test with a predetermined sample size and a strict analysis plan.

Prerequisites

Before you begin, you must have the following:

• An Optimizely Analytics account.
• A flag in Feature Experimentation. See Manage flags.

Create a Frequentist (Fixed Horizon) A/B test rule

Create an A/B test rule

1. Select a flag from your Feature Experimentation project.
2. Create an A/B test rule in your desired environment. See Run A/B tests in Feature Experimentation.

Configure your A/B test rule

1. Configure your rule as normal.
2. Complete the following for Stats Configuration:

   1. Select Frequentist (Fixed Horizon) for the Stats Configuration. This replaces the default Sequential (Optimizely Stats Engine) statistical methods with the Frequentist (Fixed Horizon) approach. See Statistical analysis methods overview for information on the differences between Sequential (Stats Engine) and Frequentist (Fixed Horizon) testing.

      

   2. (Optional) Update the Statistical Significance Level. The statistical significance level defaults to what is set in the project's Settings. See Change the statistical significance setting in Optimizely Experimentation for information.

      You should set the statistical significance level to 90% or 95%. The significance level must be between 70% and 99%. 

      The significance level set here must match the value you use in the Frequentist (Fixed Horizon) Sample Size Calculator in the next step in this configuration.

Use the Frequentist (Fixed Horizon) Sample Size Calculator

Use the Sample Size Calculator to determine the number of visitors you need in each variation before you can analyze the results.

Click Sample Size Calculator and enter the following information:

• Metric Type
• Baseline Metric Value
• (Optional) Metric Variance
• Minimum Detectable Effect
• Statistical Significance Level
• Number of Variations

Metric Type

Select the type of metric you want to analyze from the Metric Type drop-down list.

• Conversion Metrics – Binary metrics that track whether an event happened, such as a visitor converting or not. Examples include sign-ups or completed purchases.
• Numeric Metrics – Metrics that measure continuous values, such as revenue, items added to a cart, or total clicks per visitor.

Baseline Metric Value

Enter the current value of the metric you are testing, before any experiment changes. 

The Baseline Metric Value is the starting point or current value of the metric you are testing before making any changes in your experiment. It represents how your metric is currently performing with no variations applied.

For example, if your sign-up rate is currently 5%, enter 5.

Metric Variance (Optional)

If you selected Numeric for the Metric Type, enter the Metric Variance.

Variance reflects the variation in metric values. A higher variance often means that more data may be needed to detect the treatment effect.

To calculate variance yourself, use the following formula:

\[ \text{sample variance} = \frac{\sum_{i=1}^{n}(x_i - \bar{x})^2}{n-1} \]

The following is a step-by-step example using example data to calculate variance for numeric metrics:

For example, your metric is "minutes spent in app per user" for five users, with values 4, 6, 5, 9, 6.

1. Find the average of the data. \[ \frac{4 + 6 + 5 + 9 + 6}{5} = 6 \]
2. For each data point, subtract the average to see how far it is from the mean. \[ 4−6 = −2 \] \[ 6−6 = 0 \] \[ 5−6 = −1 \] \[ 9−6 = 3 \] \[ 6−6 = 0 \]
3. Square each of those differences. \[ (−2)^2 = 4 \] \[ (0)^2 = 0 \] \[ (−1)^2 = 1 \] \[ (3)^2 = 9 \] \[ (0)^2 = 0 \]
4. Add up all the squared differences. \[ 4 + 0 + 1 + 9 + 0 = 14 \]
5. Divide that total by (n − 1), where n is the number of data points. \[ \frac{14}{5 - 1} = \frac{14}{4} = 3.5 \]

Minimum Detectable Effect (MDE)

Enter the Minimum Detectable Effect. The MDE is the smallest change in your metric that you want to detect with statistical confidence. 

Because Optimizely reports and tests relative improvement, the MDE is also defined in terms of relative improvement rather than absolute improvement. Thus, use relative improvement, not absolute improvement.

For example, detecting a 10% increase on a 5% baseline means you should enter 10%.

See Use minimum detectable effect when designing an experiment.

Statistical Significance Level

Enter the Statistical Significance Level. Ensure you use the same Statistical Significance Level you set in step two in the Configure your A/B test rule section.

Number of Variations

Enter the Number of Variations, including the baseline variation.

The Visitors needed per variation updates automatically. Click Copy value to clipboard.

Complete the Frequentist (Fixed horizon) configuration

1. Paste the Visitors needed per variation from the Use the Sample Size Calculator section in the Sample Size per Variation field. 

2. (Optional) Click + Minimum duration. Enter a whole number of days the experiment should run. 

   • The Minimum duration must be at least one day.
   • The Minimum duration is optional, but it is available if you believe that your experiment will reach its sample size before completing a full business cycle. See Seasonality and traffic spikes for information on why you should run an experiment at least one business cycle.
   • Click Clear minimum duration to remove the field.

   

Complete your A/B test rule configuration

1. Configure the remainder of your A/B test rule.
2. Click Save.
3. Start your A/B test rule and run your flag to launch your experiment. See Manage rules.

You cannot pause a Frequentist (Fixed Horizon) A/B test, but you can stop it by updating the status to Conclude. If you try to conclude the A/B test rule before the sample size and minimum duration (if set) are complete, a warning displays, but you can still conclude the rule. See Conclude rule.  

You cannot edit the Stats Configuration (including engine type, sample size, and duration) after starting an A/B test in Feature Experimentation. You must duplicate and start a new A/B test rule to update these settings. 

Wait for completion

When you run an A/B test rule using Frequentist (Fixed Horizon) as your Stats Configuration, Optimizely checks at scheduled intervals to determine if the required sample size or minimum duration (if set) is met. Because these checks happen on a schedule, the actual number of samples collected per variation may be higher than the sample size you set in the A/B test configuration.

This extra data collection is expected behavior and is statistically valid. It does not affect the integrity of your results. Optimizely calculates and displays results based on the actual number of samples collected. This approach is standard practice in Frequentist (Fixed Horizon) testing, especially in high-traffic environments.

Optimizely does not display the Frequentist (Fixed Horizon) A/B test's confidence intervals and statistical significance until the required sample size or minimum duration (if set) is met. Instead, the results page displays the metric performance per variation during the test.