Mutually exclusive experiments in Optimizely Web Experimentation and Optimizely Performance Edge

Optimizely Experimentation lets you make experiments mutually exclusive when running several experiments that are related at the same time. For example, you may want to run two experiments on the same algorithm. Visitors who see both experiments when they visit your site might behave differently from visitors who see only one of these experiments in isolation. This difference in behavior has the potential to affect your results.

To avoid this, you can make your experiments mutually exclusive. Any visitor who sees Experiment A does not see Experiment B, and vice versa. Mutually exclusive experiments help you remove noise and bias from your results to make accurate inferences based on your visitors' behavior.

When to use mutually exclusive experiments

To decide whether you should create mutually exclusive experiments, decide how likely you are to see interaction effects on the experiment. You are more likely to see these effects if:

• You are running two experiments on the same page.

• You are running two experiments on the same flow where there will likely be strong overlap from one page to the next (for example, a checkout funnel or multi-step form).

• You are running two experiments that are better to run together, such as a multivariate experiment, to see the potential interaction.

If your tests run on different areas of your site or track different goals, they are less likely to have an interaction effect.

However, in some cases, you should create mutually exclusive experiments or run sequential tests (waiting for one to end for the next to start).

• When you are testing elements on the same page with the same goal – Consider whether to run these experiments simultaneously, sequentially, or test all the element changes in the same multivariate experiment to see individual and combined performance.

• When you are testing subsequent pages of the funnel with the same goal – Consider whether to run these experiments simultaneously, sequentially (starting from the bottom of the funnel and moving up) or as a multi-page experiment.

Some organizations run simultaneous, mutually exclusive experiments to preserve data integrity. However, working with mutually exclusive experiments increases the effort involved in implementation and can significantly slow down your testing cycle.

Create an exclusion group

In Optimizely Web Experimentation and Optimizely Performance Edge, you create exclusion groups to make multiple experiments and campaigns mutually exclusive. 

1. Go to Experiments and select the Exclusion Groups tab.

2. Click Create New Exclusion Group.
   

3. Name the group (required) and enter a description, if desired.

4. Add any current experiments, if desired.

5. Click Create Exclusion Group.

Mutually exclusive experiments and exclusion groups

Optimizely Web Experimentation and Optimizely Performance Edge use a unique value for each experiment to bucket each visitor. When experiments are not mutually exclusive, visitors are eligible for every experiment they are not in the holdback.

In mutually exclusive experiments, Optimizely creates a unique value for the visitor so that they are only bucketed into one experiment. Because the values are random and unique, some visitors enter multiple experiments. If visitors who see both Experiments A and B behave differently from visitors who see only Experiment A or B, then the results for A and B are skewed by the overlap. This is called an interaction effect.

If Experiments A and B are mutually exclusive, Optimizely chooses a random value (unique to the exclusion group) to bucket visitors into either Experiment A or Experiment B. This method ensures visitors do not see multiple experiments from each exclusion group.

The following sections show descriptions and diagrams to explain mutually exclusive experiments and exclusion groups in more detail.

Overlapping experiments with 100% traffic allocation

If you have two overlapping experiments, each with 100% traffic allocation, all visitors are bucketed into both experiments (as long as they meet the target conditions).

Overlapping experiments with different traffic allocation

Suppose you have two overlapping experiments, but they each have different traffic allocations

• Experiment A – 100% traffic allocation.

• Experiment B – 50% traffic allocation.

All visitors see Experiment A. About half of the visitors only see Experiment B, and about half see both Experiments A and B. Some visitors do not see Experiment B.

One overlapping experiment with two mutually exclusive experiments, 100% traffic allocation

Suppose you have three experiments, but two are in an exclusion group

• Experiment A – 100% traffic allocation.

• Experiment B – 50% traffic allocation and in Exclusion Group 1.

• Experiment C – 50% traffic allocation and in Exclusion Group 1.

All visitors see Experiment A. Some visitors see Experiment B, and some see Experiment C. However, no visitors see both Experiment B and Experiment C. Experiments B and C are in the same exclusion group, which makes them mutually exclusive.

One overlapping experiment with two mutually exclusive experiments, some traffic not allocated

In this example, you have three experiments, two mutually exclusive, but some of your traffic is not allocated to either mutually exclusive experiment

• Experiment A – 100% traffic allocation.

• Experiment B – 20% traffic allocation and in Exclusion Group 1.

• Experiment C – 50% traffic allocation and in Exclusion Group 1.

  • Some traffic (30%) in Exclusion Group 1 is not allocated to either Experiment B or C

All visitors see Experiment A. Experiments B and C are still mutually exclusive, so some visitors see Experiment B, some see Experiment C, and none see both. However, in this case, 30% of visitors see only Experiment A because 30% of visitors are not allocated to either experiment in the exclusion group.

Experiments in two exclusion groups

You can also use two different exclusion groups to add layers of mutually exclusive experiments. For example

• Experiment A – 100% traffic allocation.

• Experiment B – 20% traffic allocation and in Exclusion Group 1.

• Experiment C – 50% traffic allocation and in Exclusion Group 1.

  • Some traffic (30%) in Exclusion Group 1 is not allocated to either Experiment B or C.

• Experiment D – 50% traffic allocation and in Exclusion Group 2.

• Experiment E – 50% traffic allocation and in Exclusion Group 2.

All visitors see Experiment A, some visitors see Experiment B, some see Experiment C, and none see both B and C. Again, 30% of visitors do not see either Experiment B or C.

In this case, because of Exclusion Group 2, you have more possibilities. All visitors see either Experiment D or E, but none see both. A visitor could see experiments from all three layers: Experiment A, plus one experiment from each of Exclusion Group 1 and Exclusion Group 2. It is also possible for a visitor to see experiments from only two layers because some traffic is not allocated to either experiment in Exclusion Group 1.

Paused experiments

Pausing an experiment no longer buckets new visitors into that experiment.

Visitors who have already seen the experiment remain bucketed into the experiment, although they do not see it again for as long as it is paused. If you resume the experiment, returning visitors who are bucketed into the experiment begin seeing it again, and new visitors could be bucketed into it.

For example, a visitor is bucketed into Experiment C when they visit your site. If you pause Experiment C, the visitor does not see it anymore. If you resume the experiment, and the visitor returns to your site, they see Experiment C again because they are still bucketed into it.

Personalization campaigns and exclusion groups

You can also add personalization campaigns to exclusion groups to make experiments and campaigns mutually exclusive. In the example shown below, all visitors see Experiment A. Some visitors also see either Experiment B or Campaign A. Some visitors see only Experiment A.

When you add a Personalization campaign to an exclusion group, the campaign's holdback is not affected.

Two experiments that target different pages

When two mutually exclusive experiments target different pages, the exclusion group's decision on which experiment to bucket a visitor in is sticky and deterministic. If a visitor is bucketed into Experiment A on your site's homepage, and Experiment A is in the same exclusion group as Experiment B on your site's order page, the visitor does not see Experiment B.

Two experiments that target non-mutually exclusive and mutually exclusive audiences

When two mutually exclusive experiments target different audiences that are not mutually exclusive, the exclusion group decision about which experiment to bucket a visitor in is sticky and deterministic. For example, you have two non-mutually exclusive audiences, like "visitors in California" and "all visitors." If a visitor from the "visitors in California" audience is bucketed into Experiment A on your site's homepage, and Experiment A is in the same exclusion group as Experiment B, the visitor does not see Experiment B, even though they also qualify for the the "all visitors" audience.

The same is true when the two mutually exclusive experiments target different pages and non-mutually exclusive audiences and when you have three mutually exclusive experiments that target the same page.

Two mutually exclusive experiments that target different mutually exclusive audiences, like "Android users" and "iPhone users," are also sticky and deterministic. However, if the audiences for your experiments are mutually exclusive, you do not need to use exclusion groups for the experiments themselves.

Two experiments with different traffic allocation ratios

If an experiment is in an exclusion group, 100% of that experiment's traffic is allocated to the percentage you have allocated to the exclusion group. For example, your Exclusion Group 1 includes two experiments with these traffic allocations:

• Experiment A – 70% traffic allocation

• Experiment B – 30% traffic allocation

100% of the traffic you send to Experiment A (70% of traffic to Exclusion Group 1) sees Experiment A.

Removing experiments and campaigns from exclusion groups

If you remove an experiment or campaign from an exclusion group, visitors who were previously bucketed into that experiment or campaign are dropped from the exclusion group. Visitors do not see any other experiments or campaigns in that exclusion group. If the experiment or campaign is still running and the visitors meet the targeting conditions, they may still see the experiment or campaign.

Duplicating experiments and campaigns in an exclusion group

If you duplicate an experiment or campaign that is part of an exclusion group, the new experiment or campaign will not be associated with any exclusion group.

Archiving exclusion groups

Before you archive an exclusion group, remove all experiments and campaigns. You cannot archive exclusion groups with experiments or campaigns in them.

Visitor bucketing IDs

Visitors have a consistent bucketing ID when they return to the same exclusion group, experiment, or campaign to be bucketed. This bucket ID is different for a visitor across exclusion groups, experiments, or campaigns.

Best practices

• Make experiments mutually exclusive only when required to prevent restricting traffic more than necessary.
• Experiments within an exclusion group do not need to start or stop simultaneously, but they must all be part of the exclusion group from the moment the first experiment starts until the last experiment finishes to maintain a fixed traffic allocation and visitor bucketing.
• No experiments should be added to the group after one has been started. If experiments are added to the group while any of the group's existing experiments are already running, traffic allocation to the experiments will be shifted, and visitors may be exposed to multiple experiments after all.
  • For example, a visitor is initially bucketed into experiment A and has already been exposed to it. Then, experiment B gets added to the exclusion group. Traffic allocation will shift, and this visitor may now be assigned to experiment B while they no longer see experiment A.
• Do not remove running experiments from an exclusion group or reallocate traffic in ways that could introduce bias or overlap between experiments.
  • For example, an exclusion group contains three experiments. A visitor is initially bucketed into experiment B. Then, experiment C is removed from the group, leaving only experiments A and B. With the bucketing ranges now being shifted, the visitor may get bucketed from experiment B into experiment A.

Debugging tips

Use the JavaScript API and console log for debugging:

?optimizely_log=debug

Look for Group in the logs.

You can also check window.optimizely.get('data').groups on a page that has a grouped experiment or campaign running on it. The weightDistributions property contains a list of entities and their campaign IDs.

If there is at least one experiment running in the exclusion group, all experiments are displayed in the Optimizely log, including paused and draft experiments. The snippet does not include or run paused and draft experiments; however, paused and draft IDs are present in the exclusion group decision. A visitor could be allocated to a draft experiment before publication.

If you allocate anything less than 100% of your traffic to an experiment, visitors included in that unallocated percentage do not see any experiment in the exclusion group. If the exclusion group allocation is later increased to include 100% of incoming traffic, visitors who saw no experiment are evaluated into the experiment that filled the missing percentage.

A value of none in the Optimizely log for the group object represents the allocation to the percent of traffic not allocated to an experiment. This only occurs when a mutually exclusive group does not allocate 100% of visitors to an experiment.