To continue making our product better and easier to use for everyone, we have made a slight UI update to communicate better how Optimizely's Stats Engine handles outliers in your revenue metrics. There have not been any underlying changes in our Stats Engine or how Optimizely calculates results. This update is purely a phrasing enhancement to improve our user's understanding of what happens to their outlier values.
What is an outlier?
An outlier is an abnormally lower or higher value than other values in your results.
What are Revenue Metrics?
In Optimizely, a metric is a quantitative measurement of a visitor's action. Metrics are created out of events, which directly track actions like clicks, page views, form submissions, purchases, and scroll depth.
The total revenue metric tracks the total revenue generated from user interactions with your event. Revenue usually is not your primary metric because, unlike clicks or page views, revenue does not measure a specific, discrete action taken by your visitors. However, tracking revenue is a fantastic way to tie your optimization efforts to the metrics that your company values most.
What is changing?
We are only updating the Optimizely UI and documentation to read "Outlier Smoothing" instead of "Outlier Filtering." We believe this update to the phrasing will help users understand what is happening with their outlier values.
Why smooth outliers in revenue metrics?
Outliers can severely skew the accuracy of any analysis conducted on a data set and can lead to potentially incorrect conclusions. Smoothing outliers results in more accurate findings. We recommend turning on this setting for your revenue metric to improve the integrity of your results.
How does Optimizely treat outliers?
For revenue metrics, Optimizely gives you the option to use outlier smoothing. Optimizely's outlier smoothing algorithm first identifies any values exceeding the daily exclusion threshold, extreme values three standard deviations higher than the observed mean. These extreme values are designated as outliers. Next, Optimizely replaces these outliers with the metric's harmonic mean value. This step in the process is known as outlier smoothing. Optimizely recalculates the daily exclusion threshold for each day using a moving average of your metric's arithmetic mean and standard deviation over the previous seven (7) days. This process repeats each day of the experiment. Please view our documentation on how Optimizely handles outliers for more detailed information.
Conclusion
If you have any questions or feedback, feel free to email us at support@optimizely.com! Keep on optimizing!