What is CUPED?
CUPED (Controlled Experiment Using Pre-Experiment Data) is a statistical method that reduces variance in A/B tests, enhancing their sensitivity and making it easier to detect differences between groups. By lowering variance, CUPED enables experiments to achieve statistical significance with less data, provided that there is true treatment effect. Introducing CUPED as an option for Optimizely A/B/n tests will address challenges like insufficient traffic and high variance, making experiments more efficient.
Here are some important considerations to keep in mind when using this feature. CUPED can help reduce variance, which may enhance the significance of experiment results in certain cases.
-
- Metric Compatibility: This feature is available only for numeric metrics (not conversion metrics) as it is most effective for these types of data.
- Covariate Limitation: Covariates refers to "pre-experiment data" used for adjustments. Only pre-experiment calculations of the primary and secondary target metrics are used as covariates. User-defined customization of features is not yet supported.
- Supported Platforms: Currently, this feature works exclusively on Snowflake. BigQuery support will be added soon. We welcome requests for additional warehouse support.
- Health Checks: No "health checks" are available yet for unexpected feature imbalances (similar to SRM checks). Unbalanced features, especially with sparse prior data, may increase variance, but this is an empirical issue that may not occur.
- Data Requirement: Prior data spans from two weeks before the experiment's start date to the user's first decision event. If no prior data exists (e.g., for a new metric), CUPED will have no effect.
Closed Beta: This functionality is in beta. Please contact your Customer Success Manager if interested.
How to enable CUPED?
To enable CUPED,
- Create a new Experiment Scorecard in Warehouse-Native Analytics. Learn more
- On the Scorecard definition page, choose the preferred Experiment using the selector, and enable CUPED using the toggle.
In the NetSpring Experiment Scorecard template, using CUPED versus not using CUPED impacts the variance reduction and the sensitivity of your experiment metrics. Here's what the results of the Dragon Recommendations scorecard that was built here would look like with and without CUPED.
Please sign in to leave a comment.