Take action now: SRM detected experiment health

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Performance Edge
  • Optimizely Feature Experimentation
  • Optimizely Full Stack (Legacy)

A SRM detected experiment health on the Optimizely Experiment Results page means that your A/B test has failed a traffic imbalance check. The patterns of visitor assignments to variations in your experiment must be investigated to identify the problems. 

When you launch a Stats Engine A/B test, Optimizely Experimentation's automatic sample ratio mismatch (SRM) detection runs in the background. This algorithm identifies imbalances in visitor traffic to an experiment's variations, so you know of potential problems early.

An experiment displaying a SRM detected health status indicates that Optimizely Experimentation's automatic SRM detection algorithm identified a significant imbalance. The algorithm checks solely on the first decisions, which includes just the first decision (impression) for a visitor either within a variation (for visitor-scoped metrics) or session (for session-scoped metrics). See How Optimizely Experimentation counts conversions.

Identifying an SRM among these first decisions signals a critical imbalance. It indicates an issue in how visitors are initially bucketed, which can impact the accuracy of experiment results.

Because Optimizely's SRM detection can uncover a potential SRM earlier than a traditional chi-squared test, it is important to examine the data on the specific date when SRM is identified.

SRM detected health status

A SRM detected experiment health status is statistically significant, and there is evidence of consistent and undeniable underlying assignment bias. The experiment should be paused immediately to investigate the issue.

See Imbalance detected: What to do if Optimizely's automatic SRM detection alerts you to an imbalance in your Stats Engine A/B test. 

Why an SRM occurred

For a list of possible reasons, see Possible causes for traffic imbalances.