Holdback: Measure overall impact in Personalization

Every Optimizely Web Personalization campaign has a holdback, like the control of an A/B test. Optimizely Web Experimentation's Stats Engine uses the holdback to help you track how well your personalized experiences perform compared to the generic experience.

By default, Optimizely Web Personalization shows a personalized experience to 95% of visitors in a campaign and holds back 5% who will see the original.

Visitors are bucketed into the holdback by chance.

Holdbacks are persistent. When a visitor is bucketed into the holdback, they always see the holdback experience for that campaign.

Managing the balance between too big of a holdback and too small can be confusing. Five percent holdback is the industry standard and is what Optimizely recommends. Five percent is a good balance and lets you monitor how well your personalization is doing.

A larger holdback will not necessarily help you gather data about statistical significance faster. Instead, with a giant holdback, you risk underpowering your experiment. An experiment that is too underpowered may cause it to be inconclusive. Remember, your traffic is finite and having a large holdback will cause a large amount of that traffic not to see the personalized experience. 

Optimizely Web Experimentation integrates testing and personalization, so you can be confident that the experiences you deploy generate lift. In Optimizely Web Personalization, consider directing the majority of your traffic to the personalized experience (versus the holdback) to maximize lift.