Holdback: Measure overall impact in Personalization

  • Updated

This topic describes how to:

  • Track how well-personalized experiences perform compared to the original
  • Set or change the holdback

Every Optimizely Web Personalization campaign has a holdback, like the control of an A/B test. Optimizely Web Experimentation's Stats Engine uses the holdback to help you track how well your personalized experiences perform compared to the generic experience.

By default, Optimizely Web Personalization shows a personalized experience to 95% of visitors in a campaign and holds back 5% who will see the original.

Visitors are bucketed into the holdback by chance.

Holdbacks are persistent. When a visitor is bucketed into the holdback, they always see the holdback experience for that campaign.

Managing the balance between too big of a holdback and too small can be confusing. Five percent holdback is the industry standard and is what Optimizely recommends. Five percent is a good balance and lets you monitor how well your personalization is doing.

A larger holdback will not necessarily help you gather data about statistical significance faster. Instead, with a giant holdback, you risk underpowering your experiment. An experiment that is too underpowered may cause it to be inconclusive. Remember, your traffic is finite and having a large holdback will cause a large amount of that traffic not to see the personalized experience. 

Optimizely Web Experimentation integrates testing and personalization, so you can be confident that the experiences you deploy generate lift. In Optimizely Web Personalization, consider directing the majority of your traffic to the personalized experience (versus the holdback) to maximize lift. 

Set the holdback

When you create a campaign, set the holdback in the final step.


Change the holdback

Stats Engine does not currently support changing the holdback for a running campaign. If you choose to do this, you may see invalid statistical reports.

Optimizely Web Experimentation sets one holdback per campaign. So, the holdback will stay the same no matter how many experiences or variations you add to a campaign.

To change the holdback for an existing campaign:


  1. Navigate to the Experiments dashboard and select the campaign.

  2. In the Manage Campaign dashboard, click Settings.

  3. Edit the holdback.

  4. Click Save.

To reset the holdback, click Revert instead of Save.