My experiment is reporting a minor imbalance for its experiment health status. How worried should I be?

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Performance Edge
  • Optimizely Feature Experimentation
  • Optimizely Full Stack (Legacy)

The bottom line

That experiment's health status listed in the results page sidebar alerts you to unusual patterns of visitors assigned to your experiment.

When Experiment Health on your experimentation results page displays Minor for a Stats Engine experiment, it indicates that unusual visitor assignments biased to a particular variation have gradually accumulated. Immediate action may not be required.

The yellow exclamation point

minorimbalance_nobanner_2024.png

When you launch a Stats Engine A/B test, Optimizely's automatic sample ratio mismatch (SRM) detection runs in the background. This algorithm finds unusual imbalances in visitor traffic to an experiment's variations and detects experiment problems early if they arise.

The automatic SRM detection tool combines the power of automatic visitor imbalance detection with an indicator of experiment health. This indicator alerts an experimenter to any visitor traffic issues and tells you when a critical traffic imbalance occurs.

Minor health status

A Minor experiment health status indicates that there was a gradual accumulation of slightly out-of-the-ordinary visitor counts. The algorithm observed consistently unusual patterns related to total decisions. This is different than patterns observed in first decisions, which capture when a visitor is initially bucketed into a variation and any decisions where the visitor is in the same variation are filtered out.

Steps to take for Minor health status

  1. An experiment is still valid with a Minor health status.
  2. At the conclusion of the experiment, review your experiment configuration for any implementation issues.
  3. Typical minor triggers are redirect experiments or menus firing multiple events.

See: Imbalance detected: What to do if Optimizely's automatic SRM detection alerts you to an imbalance in your Stats Engine A/B test.

Examples of why a minor imbalance may have occurred

The following examples are not an exhaustive range of minor imbalance causes. 

The following are examples of situations that may have occurred. Decide what to do with your experiment, depending on your situation. See additional Causes of imbalance.

Example 1: Redirect experiments and imbalances

When you create a redirect or URL split test, versions A and B sit on different URLs, while in a typical A/B test, all versions in the experiment have the same URL. Redirect tests can be efficient for small teams by recycling and reusing content while testing for impact. An example of this is homepage hero testing.

Redirect experiments produce valid results. However, due to the nature of redirects, users may close the window or tab and exit a page before the redirect finishes executing. Optimizely does not receive the data, which drops the user from being counted. Optimizely expects the behavior that a slight imbalance beyond a 2% fluctuation may occur.

There are multiple reasons why redirects are associated with imbalances. For example:

  1. The browser may reject it if there are too many redirects. Optimizely may not be the only thing redirecting the user; it may be one step in a series of redirects.
  2. A user may have a browser setting or extension that rejects redirects.
  3. The delay could be long enough that a user closes the tab before the redirect finishes.

Example 2: You have a highly specific targeted audience, and your experiment displays an imbalance after 48 hours.

When there are conditions and constraints on exposure to an experiment, a slight imbalance can emerge before the first business cycle of an experiment is completed (usually seven days). These experiments can offer an experiment program valid results, but it is the decision of the experimenter and their tolerance level for imbalances before the completion of one business cycle. These imbalances do resolve after one business cycle.