Global holdouts

Global holdouts help you accurately measure the true impact of your A/B testing and experimentation efforts. You designate a small percentage of users as a control group, ensuring they do not experience any experiments or new feature rollouts. This lets you directly compare outcomes between users who experience variations identified as winners during A/B tests and those who only see the default “off” variation for feature flags in your project. This approach lets you quantify the cumulative impact of your testing program, answer critical questions from leadership about the value generated, and make informed decisions about future product and marketing strategies.

A/B testing optimizes products and enhances user experiences, but proving its value can be challenging without a clear measurement of the overall uplift generated. Quantifying its revenue impact remains difficult, as measuring contributions over a quarter or year often requires manual calculations and complex workarounds.

Global holdouts solve this problem by setting aside a small percentage of their user traffic (typically up to 5%) as a control group. These users are not exposed to any A/B tests or feature rollouts, ensuring a clean baseline for comparison. The holdout group only experiences the default product experience as configured in the “off” variation of your feature flags.

Native global holdouts

Optimizely's native global holdouts provide a streamlined way to establish a persistent control group directly within the platform.

How global holdouts work

• Configuration – You create a holdout in the platform, select the environment, define the primary metric to track, and choose the percentage of traffic to hold back. The system warns you if the holdout exceeds 5% of total traffic. The more visitors you withhold from experiments, the time needed to reach statistical significance increases.
• Assignment – Visitors in the holdout group receive the default "off" variation for feature flags, regardless of ongoing A/B tests, targeted deliveries, or multi-armed bandits running on top of your flags.
• Visibility – You can monitor and analyze holdout performance through a dedicated dashboard.
• Results calculation – After experiments conclude and the winning variation for the experiment is manually identified in the application, the platform aggregates data from visitors who saw winning variations to compare against the holdout group. This lets you compare key metrics, such as revenue or engagement, to quantify the uplift generated by experimentation. See how to manually deploy a winning variation.

Create a holdout

1. Go to Flags > Holdouts.

2. Click Create Holdout.

   

3. Enter a unique name in the Name field.
4. Edit the Holdout Key and optionally add a Description. Valid keys contain alphanumeric characters, hyphens, and underscores, are limited to 64 characters, and cannot contain spaces. You cannot modify the key after you create the holdout.
5. Select the Environment to scope the holdout. Like A/B tests, holdout results are separated by environment.
6. Choose the primary Metric. This is usually the main company metric to measure results. Optionally, you can choose secondary and monitoring metrics.
7. Set the percentage of traffic to be held back by Feature Experimentation from tests and new features (recommended not to exceed 5%). The system warns you if the holdout percentage exceeds 5%.

8. (Optional) Target a specific audience, although you should apply the holdout to everyone.

   

9. Click Create Holdout.

Start a holdout

After creating the holdout, it displays in the Holdouts tab, where you can start it.

Click More (...) for the holdout and click Start.

Visitors assigned to this holdout only see the default "off" variation for any flags within the holdout, regardless of any experiments they might otherwise be included in by Feature Experimentation. When you start a holdout, you cannot pause it; only permanently conclude it.

To see holdouts that Feature Experimentation applied to a rule, go to Flags > Environment > Rule. The holdouts applied, along with the traffic allocated, display.

Results calculation

After you conclude A/B tests and identify winning variations, data from these variations is aggregated and compared to the holdout group’s data.

The comparison provides a clear measurement of uplift and the cumulative impact of experimentation, similar to a large-scale A/B test.

Learn how to conclude your A/B test and deploy the winning variation.

You must conclude your A/B test and deploy the winning variation to reflect your experiment results in the holdout analysis accurately. This step confirms the variation is part of the exposed group and includes its performance in the holdout results.

Holdout results compare unique conversions per visitor and the conversion rate of users bucketed in the holdout with those that were exposed to each winning variation, as selected by the customer. The comparison between the In Holdout and Not in Holdout conversion rates generates the confidence interval, statistical significance calculation, and cumulative improvement percentage.

Conclusion

Global holdouts help you measure the success of your experimentation programs by maintaining a persistent control group and providing transparent, actionable results. They empower you to maximize the value of your A/B testing efforts and make data-driven decisions for growth. Additionally, they offer a robust methodology for long-term measurement of product changes and feature rollouts, letting you quantify the value of experimentation and justify investment in A/B testing.

Self-service global holdout

Self-service global holdouts let you define a control group using custom logic, user attributes, and audiences. This approach is flexible but requires more manual configuration.

Self-service global holdout groups are the following:

• Utilized only during a specified timeframe.
• Comprised of a set percentage or a fixed portion of traffic.
• Implemented to assess the cumulative impact of experiments by comparing users in the control group to those in test variants.

Feature Experimentation is stateless and uses deterministic bucketing to make decisions. Because of this, global holdout groups are only directly supported out of the box using the stateless global holdout strategy. You can also define a global holdout with a persistent data store using the stateful global holdout strategy. However, you need a third-party or self-hosted infrastructure to implement this strategy.

Each strategy has its advantages and drawbacks, so evaluating which approach best aligns with your specific needs and architecture capabilities is important. No matter what solution you choose, you must include a default variation on each flag for users in the global holdout group to receive.

Stateless global holdout (recommended)

The easiest way to create a global holdout group is to create a user attribute and audience and add that audience to all experiments.

Pros

• Does not require third-party or separate hosted infrastructure.
• Naturally scales as the user base grows (which can account for seasonality effects).
• Is randomly sampled.

Cons

• Is not static, requiring data analysts to know how the group is defined to determine whether a user is a member.

Example implementation steps

Example 1 – Create a 5% global holdout.

This example uses the last two digits of the user id to determine if the user is in the global holdout.

1. Create a user attribute named lastTwoOfId that you can use to determine if a user is in a global holdout group. See Define attributes to create attributes in the UI or the Create an Attribute endpoint in the API reference documentation.
2. Create an audience named Not in Global Holdout. See Target audiences for the UI or Create an Audience endpoint.
3. Drag and drop the lastTwoOfId attribute.

4. Configure the Audience Conditions drop-down lists to check if the Visitor matches the lastTwoOfId attribute where Number is greater than 4. 

   

5. Add the Not in Global Holdout audience to all experiments using an and condition with any other audiences you want to target to ensure the experiment considers only users who are not in the global holdout group.
6. After you pass the data into the user attribute, the previous steps create a global holdout group. Users in that group receive the default variation for each flag because they do not qualify for the Not in Global Holdout audience.

Example 2 – Use a boolean to determine if the user is in the global holdout group.

1. Create an attribute named inGlobalHoldout that you can use to determine if a user is in a global holdout group.
2. Use your own logic at the SDK level to determine if the user is part of the global holdout group and pass true as the value for the boolean user attribute inGlobalHoldout.
3. Add the Not in Global Holdout audience to all experiments using an and condition with any other audiences you want to target to ensure the experiment considers only users who are not in the global holdout group.

After you pass data into the new user attribute, the previous steps create a global holdout group. Users in that group receive the default variation for each flag because they do not qualify for the Not in Global Holdout audience.

Stateful global holdout

Instead of relying solely on user attributes, the stateful approach uses a dedicated experiment to determine holdout membership and dynamically applies that decision across all other experiments.

You can create a static list of users to comprise a global holdout group. You must persist that data using a third-party or self-hosted data store and pass it into a user attribute.

Pros

• A data analytics team, for example, can statically define the global holdout in advance and does not grow unless edited.
• You can define users in the global holdout group in a way that avoids bias (or introduces it, so be careful).
• It may be easier to query for global holdout membership with a persisted datastore, for example, in a data lake.

Cons

• Third-party or customer-hosted infrastructure is required for persisting global holdout membership, as it is not defined deterministically (on read).
• Developer effort is required to safely expose the static list of global holdout member IDs to all Feature Experimentation SDK implementations, like through an API, and implement the new user attribute.
• Assuming the user base grows while the global holdout group remains static, the global holdout group shrinks proportionally.

Example implementation steps

Expose a static list of global holdouts in a way accessible to all Feature Experimentation SDK implementations. First, create and run an experiment to determine if the user is in the global holdout group. Assign users to an attribute and use this attribute in future experiments to filter out users from the holdout group.

1. Create a global holdout A/B test flag. This experiment determines whether a user is in the holdout. Evaluate this flag as early as possible in the user journey (for example, at login or first visit).
   • 5% of users are assigned to holdout (do not see any experiments).

   • 95% of users proceed as normal (eligible for experiments). 

     

2. Run Global holdout flag before any new experiment. Then create an attribute named eligibleForTests based on the global holdout experiment result. Which variation the user receives determines if they are eligible for the experiment.
   • On variation – User is eligible for experiments.

   • Off variation – User is in holdout and should not see any experiments. 

     

3. Create an audience named Eligible for tests using the eligibleForTests attribute as an audience filter.

   • Configure the Audience Conditions drop-down lists to check if the Visitor matches the eligibleForTests equals on. If the user is in the On variation, they are eligible for future experiments. 

     

4. Use the Eligible for tests audience for any future flags. This audience filters out users who are in the global holdout.

For example, create an A/B test flag rule named experiment_flag that uses the Eligible for tests audience, like the following: 

The following JavaScript sample code demonstrates how to call the decide() method on the global_holdout_experiment flag and set the eligibleForTests attribute based on the decision. Then, it retrieves a decision for experiment_flag, which uses eligibleForTests as an audience condition.

const user = optimizely.createUserContext("user123");

/* -- Make decision for global holdout first -- */
const holdoutDecision = user.decide("global_holdout_experiment");

user.setAttribute("eligibleForTests", holdoutDecision.variationKey);

/* -- If eligibleForTests=off for the user, they do not qualify for any experiment audiences -- */
experimentDecision = user.decide("experiment_flag");

Comparison between native Global holdouts and Self-serve options