Holdback: Measure overall impact in Personalization

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Web Personalization

Every Optimizely Web Personalization campaign has a holdback, like the control of an A/B test. Optimizely Web Experimentation's Stats Engine uses the holdback to help you track how well your personalized experiences perform compared to the generic experience.

By default, Optimizely Web Personalization shows a personalized experience to 95% of visitors in a campaign and holds back 5% who will see the original.

Visitors are bucketed into the holdback by chance.

Holdbacks are persistent. When a visitor is bucketed into the holdback, they always see the holdback experience for that campaign.

targeting-25.png

Managing the balance between too big of a holdback and too small can be confusing. Five percent holdback is the industry standard and is what Optimizely recommends. Five percent is a good balance and lets you monitor how well your personalization is doing.

A larger holdback will not necessarily help you gather data about statistical significance faster. Instead, with a giant holdback, you risk underpowering your experiment. An experiment that is too underpowered may cause it to be inconclusive. Remember, your traffic is finite and having a large holdback causes a large amount of that traffic not to see the personalized experience.

Optimizely Web Experimentation integrates testing and personalization, so you can be confident that the experiences you deploy generate lift. In Optimizely Web Personalization, consider directing the majority of your traffic to the personalized experience (versus the holdback) to maximize lift. 

Set the holdback

When you create a campaign, set the holdback in the final step.

Stats Engine does not currently support changing the holdback for a running campaign. If you choose to do this, you may see invalid statistical reports.

Optimizely Web Experimentation sets one holdback per campaign. So, the holdback stays the same no matter how many experiences or variations you add to a campaign.

To change the holdback for an existing campaign:

  1. Go to the Experiments dashboard and select the campaign.
  2. In the Manage Campaign dashboard, click Settings.
  3. Edit the holdback.
    targeting-26.png
  4. Click Save.

To reset the holdback, click Revert.