Outlier management

The New A/B Results page is currently in beta. Contact your Customer Success Manager or sign up for the beta on Optimizely.com.

Outliers are extreme values that can distort that can distort your experiment results. A few extreme values pull the average off-center, inflate variance, and delay a statistically significant result. Outlier management caps these values to make them less extreme, which reduces their effect on your results. You can configure outlier management on a per-metric level.

Outlier management methods

Optimizely lets you choose between two outlier management methods: metric capping (capping at a specified metric value) and Winsorization (capping at a specified percentile). On the results page, these options are called constant and percentile, respectively.

Winsorization (percentile option)

Winsorization caps extreme values using a percentile threshold. It replaces every value above a percentile you choose with the value at that percentile, which Optimizely calculates automatically from your data. For example, at the 99th percentile, Optimizely takes all values above the 99th percentile (roughly the top 1% of your data) and lowers each of them to the 99th-percentile value. The other 99% of values are left unchanged, so only the extreme upper tail is affected.

Winsorization caps outliers rather than discarding them. It keeps your full sample size while limiting the influence of a few extreme values. Optimizely uses the 99th percentile as the default for Winsorization. Winsorization is a good fit when you are not sure about a specific data point you need to cap at.

Metric capping (constant option)

Metric capping caps values at a fixed constant you set. It replaces every value above that constant with the constant itself. For example, if you set the constant to $500, Optimizely records every value above $500 as $500. Values at or below $500 stay unchanged.

Like Winsorization, metric capping keeps your full sample size and limits the influence of extreme values on your metric. The difference is that the threshold is a fixed number you choose rather than one calculated from your data. Metric capping is a good fit when you know exactly the data point you would like to cap at.

Adjustment levels

You can apply outlier management at the user level or the event level. The chosen adjustment level determines whether Optimizely applies the selected outlier management logic for each event or aggregate per user before applying the outlier logic.

Configure outlier management

Configure outlier management in a metric's advanced settings on the results page. The setting applies only to the selected metric.

1. Select the numeric metric you want to configure and expand Advanced Settings.
2. Toggle Outlier management on.
3. Select percentile or constant from the type drop-down list. Enter the threshold value:
   • For percentile, enter 90 or higher.

   • For constant, enter the maximum allowed metric value.

     

4. Use the level drop-down list to apply outlier management at the user level or the event level.

   

5. Click Run to apply the change and recalculate the results.

Example of outlier management logic

Consider an ecommerce experiment with metric capping set to $500. Two shoppers make these purchases during the test:

• Kate – $200 and $600
• Josh – $800

The adjustment level changes the total:

• User level – Optimizely sums each user's purchases, then caps the sum at $500. Kate's $800 total caps to $500, and Josh's $800 caps to $500, for a combined total of $1,000.
• Event level – Optimizely caps each purchase at $500, then sums. Kate's purchases become $200 and $500, and Josh's becomes $500, for a combined total of $1,200.