Take action now: Critical imbalance experiment health

The bottom line 

A Critical Experiment Health warning on the experimentation results page means that your experiment has failed two different imbalance checks. The patterns of visitor assignments to variations in your experiment must be investigated to identify the problems. 

The red exclamation point

When you launch a Stats Engine A/B test, Optimizely Experimentation's automatic sample ratio mismatch (SRM) detection runs in the background. This algorithm finds unusual imbalances in visitor traffic to an experiment's variations and detects experiment problems early if they arise.

Optimizely's automatic SRM detection tool combines the power of automatic visitor imbalance detection with an indicator of experiment health. This indicator alerts an experimenter to any visitor traffic issues and tells you when a critical traffic imbalance occurs.

An experiment that shows Critical health status has failed two different imbalance checks. It means Optimizely's automatic SRM detection algorithm identified an imbalance in the following:

1. Total decisions – Are the aggregated impressions per user that Optimizely Experimentation receives regardless of their variation assignment in an experiment.
2. First decisions – Reflect the first impression of a variation by a particular user, which is captured when a visitor is initially bucketed into a variation.

Critical health status

A Critical experiment health status is the highest priority visitor imbalance. It means the experiment is experiencing a statistically significant difference in traffic balance, and there is evidence of consistent and undeniable underlying assignment bias. The experiment should be paused immediately to investigate the issue.

See Imbalance detected: What to do if Optimizely's automatic SRM detection alerts you to an imbalance in your Stats Engine A/B test. 

Why a critical imbalance occurred

For a list of possible reasons, see Possible causes for traffic imbalances.