Section rollups in multivariate tests for Optimizely Web Experimentation

Interaction effects

Interactions are effects that occur only in the presence of two or more combinations together. They can be positive or negative, but either way, they make it more difficult to understand what causes the lift you see in a rollup view.

For example, suppose that you’re using a MVT to test a button’s text color in one section and the button’s background color in another section at the same time. If you test black variations for both text and background simultaneously, the combined effect will likely have a far stronger negative effect than the effects of each variation on its own.

That additional kick from the two black variations appearing together is the interaction effect.

If you had run an A/B test on text color alone, the improvement reported on the Results page would show the expected gain in conversions from changing the text color to black.

However, the section rollup A/B test improvement should not be interpreted this way. Since a large chunk of the visitors in that test came from the MVT combination showing black text against a black background, their conversion rate is much lower than we would expect to see in cases where the text is black but the background is not. And because this effect is absent in the control for the text color section, this would cause the improvement (of a black text over the control text color) reported at the rollup level to be biased downwards.

In other words, the negative response to black text on a black background will make it appear as though you should expect a smaller improvement from changing the text color to black than you are actually likely to get.

Find interaction effects in your results

To help protect yourself against interaction effects unduly influencing your business decisions, we recommend a section rollup workflow that starts by looking at MVT combinations, and proceeding to section rollups from there.

1. Examine MVT combinations for statistically significant winners

   The main reason you would use a MVT in the first place (instead of a standard A/B test) is that the MVT will give you information on potential interaction effects between sections. So the first step should always be to look at the MVT combinations. If you find statistically significant results here, use this combination as a basis for your decision, and do not proceed to the next step in this workflow.

   Otherwise, move on to the section rollup level.

2. Examine section rollups for statistically significant winners

   Interaction effects make it more difficult to interpret  results at the section rollups level. However, these interaction effects can easily be quite small, or they may even have no practical impact whatsoever. The severity of the interaction effects will determine how you interpret what’s reported on your Results page.

To determine whether strong interactions exist, start by comparing the estimated improvement from the variations in the section in question at both:

• the section rollup level, and

• the slice of the MVT containing combinations with controls in all sections except the section in question

For example, say you are working with a MVT containing three sections, each with two variations:

In this example, you want to make a decision based on the rollup for the button copy section. To determine if there are interaction effects in your sections, compare the improvement shown in the button copy section rollup to the metrics shown for combinations 1 and 5 (the combinations where both button background color and button text color use their original colors).

If the MVT is free of interactions, the estimated improvement from these two levels should be close. If they’re not—especially when the sample size within each MVT cell is large—then that should be taken as a sign that interactions are present, and you should be cautious in interpreting results at the section rollup level.

If there are no interaction effects, then the improvement reported in the section rollup is equivalent to the improvement that you would see from running this A/B test separately. The improvement you actually observe from deploying the variation should closely match the improvement reported on the results page, regardless of what variations you deploy in the other sections.

If there are interaction effects, then the improvement reported in this section rollup is essentially an average over all the candidate variations in the other sections. The improvement you actually observe from deploying the variation may or may not deviate greatly from the improvement reported on the results page, depending on exactly which combination of variations you deploy from the other sections.

View results for section rollups

To see the results of an experiment broken out on the section rollup level, follow these steps:

1. In your experiment, navigate to the Results tab.

2. From the Section dropdown menu, select the section you are interested in.

The results for the rollup you selected appear.