Local holdouts

Local holdouts extend Optimizely global holdout capabilities. A global holdout applies to one or more entire projects. A local holdout applies only to the specific rules you choose. This lets you hold traffic back from a targeted set of experiments instead of the whole project.

Local holdouts and global holdouts work independently. Creating a local holdout does not affect any existing global holdouts.

A holdout keeps a group of visitors out of your experiments. This gives you an untouched control group to measure the cumulative impact of an experimentation program. If you are new to holdouts, review the core concepts of Feature Experimentation first.

A global holdout applies to every rule in the project or projects you select. Local holdouts add precision by applying only to the rules you choose. Use them to:

• Measure the combined impact of a specific initiative, such as a set of checkout experiments, without holding traffic back from unrelated rules.
• Keep the rest of the project running at full traffic while you measure the rules in the holdout.

Local holdouts keep the same statistical rigor and automated reporting as global holdouts, while giving you control over the holdout's boundaries.

How local holdouts work

When you start a local holdout, Optimizely holds back a defined percentage of eligible visitors from the rules you selected. Held-back visitors receive the baseline experience (the default "off" flag variation) for those rules, regardless of what is running. Optimizely buckets everyone else into the rules as normal.

Optimizely aggregates results across the selected rules. It compares the holdout group against visitors who saw those rules to give you the cumulative impact of that set of rules.

Local holdouts have the following key behaviors:

• Scope is fixed at start – The set of rules in a holdout is locked when the holdout starts. This keeps how long each rule runs consistent, so results stay statistically valid.
• All-or-nothing per rule – A rule is either fully in or fully out of a holdout. You cannot include only certain variations of a rule.
• Consistent hold-back – A held-back visitor receives the baseline experience for every rule in the holdout, not only some of them.

Permissions

You can assign roles per project. See Manage collaborators in Feature Experimentation.

• Create or edit a local holdout – Editor, Collaborator, and Admin.
• View holdout results – All roles (including Viewer).
• Create audiences from a holdout – Editor, Collaborator, and Admin.

Create a local holdout

1. Go to Flags > Holdouts.
2. Click Create Holdout.
3. Enter a unique Name.

4. Edit the Holdout Key and optionally add a Description. Valid keys contain alphanumeric characters, hyphens, and underscores, are limited to 64 characters, and cannot contain spaces. You cannot modify the key after you create the holdout.

   

5. Under Scope, select the Environment this holdout applies to. See Manage environments in Feature Experimentation.
6. Select Local holdout (To hold traffic back from entire projects instead, select Global holdout).

7. Add the rules to include. Rules include A/B tests and other flag rules in the project. A local holdout must include at least one rule.

   

8. Under Metrics, click Add Metric and add one or more metrics. Metrics apply across all rules in the holdout, and the selected environment determines which metrics are available. See Choose metrics in Feature Experimentation.
9. Under Traffic Allocation, enter the percentage of visitors to withhold from each rule in the holdout. Optimizely recommends 1–5% and shows a warning if you exceed 5%.

10. (Optional) Under Audiences, search for and add audiences to apply the holdout to specific visitors. The default audience is Everyone. See Target audiences in Feature Experimentation.

    

11. Click Create Holdout.

Audience conditions set on an individual experiment take priority over audience conditions set on the holdout. For example, an audience excluded from an experiment stays excluded from it, even if the holdout does not exclude that audience.

Optimizely creates the holdout in a draft state. It does not affect traffic until you start it. See Start a holdout.

Tag and target holdout users

Local holdouts tag every user assigned to the holdout so you can reuse that group of users after the holdout ends:

• Automatic tagging – Optimizely tags each visitor in the holdout by an attribute or ID.
• Audience creation – Create an audience to Exclude holdout users or Target holdout users for follow-up experiments and analysis.
• Export – Export the list of holdout visitors for further analysis.

View holdout reports

You can open a holdout to see its results dashboard, which reports the cumulative impact of the selected experiments against the holdout group. The dashboard follows the same conventions as the Optimizely Experiment Results page. To read statistical significance and confidence intervals, see Analyze results in Feature Experimentation. The dashboard can show:

• Cumulative impact of the holdout group on your primary metrics.
• Per-experiment contribution to the overall impact.
• Trend over time for the holdout.
• Comparison against other holdouts or historical data. To break impact down by user attribute, segment your results.

Export results (including an executive summary) to PDF or CSV.

Start a holdout

After creating the holdout, it displays in the Holdouts tab, where you can start it.

Click More (...) for the holdout and click Start.

Visitors assigned to this holdout only see the default "off" variation for any flags within the holdout, regardless of any experiments they might otherwise be included in by Feature Experimentation. When you start a holdout, you cannot pause it; only permanently conclude it.

To see holdouts that Feature Experimentation applied to a rule, go to Flags > Environment > Rule. The holdouts applied, along with the traffic allocated, display.

Conclude or delete a holdout

The available actions depend on whether you have started the holdout:

• Holdout started – You cannot delete a holdout that has started. Click More (...) > Conclude instead. The holdout status changes to Concluded. Every user in the holdout is rebucketed and begins receiving variations from the running A/B Tests, Targeted Deliveries, and Multi-Armed Bandits. You cannot re-enable a concluded holdout.
• Holdout not started – Click More (...) > Delete to remove the holdout permanently from the project. The action cannot be undone.

After a holdout concludes, click More (...) > Archive when it is no longer relevant. Archives preserve the historical results and remove the holdout from the active list. To unarchive a holdout, click More (...) > Unarchive.

The deletion safeguard protects long-running holdout data from accidental loss. For example, a holdout that ran for a quarter cannot be deleted, so its results remain available for review and reporting.

Local holdouts versus global holdouts

A visitor cannot be in both a global and a local holdout at the same time. If both apply, the global holdout takes precedence. If you need to scope a holdout to specific rules, use a local holdout.

If a visitor qualifies for multiple local holdouts, the holdout that is highest in your configured priority order wins. You manage this order in the holdout settings. This is similar to how mutually exclusive experiments prevent a visitor from entering overlapping tests.

Frequently asked questions