Holdback: Measure overall impact in Personalization

  • Updated

This topic describes how to:

  • Track how well personalized experiences perform, compared to the original
  • Set or change the holdback

Every Optimizely Web Personalization campaign has a holdback, which is like the control of an A/B test. Optimizely Web Experimentation's Stats Engine uses the holdback to help you track how well your personalized experiences perform, compared to the generic experience.

By default, Optimizely Web Personalization shows 95% of visitors in a campaign a personalized experience, and holds back 5% who will see the original.

Visitors are bucketed into the holdback by chance.

Holdbacks are persistent. When a visitor is bucketed into the holdback, they always see the holdback experience for that campaign.



Make the holdback larger to get data faster, or shrink it if you are very confident. Five percent lets you monitor how well your personalization is doing.

Optimizely Web Experimentation integrates testing and personalization, so you can be confident that the experiences you deploy generate lift. In Optimizely Web Personalization, consider directing the majority of your traffic to the personalized experience (versus the holdback) to maximize lift. If you have not yet thoroughly A/B tested your experiences, consider setting a higher holdback to get significance more quickly.

Set the holdback

When you create a campaign, set the holdback in the final step.

holdback.png

A larger holdback will help you gather data about statistical significance faster, so you know the impact of your campaign. A smaller holdback allows you to show personalized experiences to more of your visitors.

Change the holdback

Changing the holdback for a running campaign is not currently supported by Stats Engine. If you choose to do this, you may see statistical reports that are invalid.

Optimizely Web Experimentation sets one holdback per campaign. So, no matter how many experiences or variations you add to a campaign, the holdback will stay the same.

To change the holdback for an existing campaign:

change-holdback.gif

  1. Navigate to the Experiments dashboard and select the campaign.

  2. In the Manage Campaign dashboard, click Settings.

  3. Edit the holdback.

  4. Click Save.

To reset the holdback, click Revert instead of Save.

revert-holdback.png