Mutually exclusive experiments in Optimizely Web Experimentation and Optimizely Performance Edge

When to use mutually exclusive experiments

To decide whether you should create mutually exclusive experiments, first ask how likely you are to see interaction effects on the experiment. You are more likely to see these effects if:

• You are running two experiments on the same page.

• You are running two experiments on the same flow where there is likely going to be strong overlap from one page to the next (for example, a checkout funnel or multi-step form).

• You are running two experiments that are actually better to run together, such as as a multivariate experiment, to see the potential interaction.

Your tests are less likely to have overlapping effects if they:

• Run on different areas of your site.

• Track different goals.

However, in some cases, we do recommend creating mutually exclusive experiments or running sequential tests (waiting for one to end in order for the next to start):

• When you are testing elements on the same page with the same goal. Consider whether to run these experiments simultaneously, sequentially, or test all the element changes in the same multivariate experiment to see individual and combined performance.

• When you are testing subsequent pages of the funnel with the same goal. Consider whether to run these experiments simultaneously, sequentially (starting from the bottom of the funnel and moving up), or as a multi-page experiment.

Some organizations choose to run simultaneous, mutually exclusive experiments, to preserve data integrity. You can accomplish this in Optimizely with the code we provide below. Note that working with mutually exclusive experiments increases the effort involved in implementation, and can significantly slow down your testing cycle.

Create an exclusion group

In Optimizely Web Experimentation and Optimizely Performance Edge, you create exclusion groups to make multiple experiments and campaigns mutually exclusive from each other. For instructions on creating exclusion groups, see the images with step-by-step instructions below.

1. Go to the Experiments dashboard and select the Exclusion Groups tab.

2. Click Create New Exclusion Group.

3. Name and describe the exclusion group.
   In this example, the exclusion group is named "Search algorithm group." Customers can add experiments on their site to this exclusion group to ensure that the experiments will be mutually exclusive. No visitor will see more than one experiment that is in this exclusion group.

4. Click Create Exclusion Group.

Mutually exclusive experiments and exclusion groups

Optimizely Web Experimentation and Optimizely Performance Edge use a unique value for each experiment to bucket each visitor. When experiments are not mutually exclusive, visitors are eligible for every experiment in which they are not in the holdback. In mutually exclusive experiments, Optimizely creates a unique value for the visitor to make sure that they are only bucketed into one experiment. Because the values are random and unique, some visitors enter multiple experiments. If visitors who see both Experiments A and B behave differently from visitors who see only Experiment A or B, then the results for A and B are skewed by the overlap. This is called an interaction effect.

If Experiments A and B are mutually exclusive, Optimizely chooses a random value (which is unique to the exclusion group) to bucket visitors into either Experiment A or Experiment B. This method ensures that visitors will not see more than one experiment from each exclusion group.

The following sections show descriptions and diagrams to explain mutually exclusive experiments and exclusion groups in more detail.

Overlapping experiments with 100% traffic allocation

If you have two overlapping experiments, each with 100% traffic allocation, all visitors will be bucketed into both experiments (as long as they meet the experiments’ targeting conditions).

Overlapping experiments with different traffic allocation

Suppose you have two overlapping experiments, but they each have a different traffic allocation:

• Experiment A: 100% traffic allocation

• Experiment B: 50% traffic allocation

In this example, all visitors will see Experiment A. About half of the visitors will only see Experiment A, and about half will see both Experiments A and B. Some visitors will not see Experiment B at all.

One overlapping experiment with two mutually exclusive experiments, 100% traffic allocation

Suppose you still have three experiments, but two are in an exclusion group:

• Experiment A: 100% traffic allocation

• Experiment B: 50% traffic allocation and in Exclusion Group 1

• Experiment C: 50% traffic allocation and in Exclusion Group 1

In this case, all visitors will see Experiment A. Some visitors will see Experiment B, and some will see Experiment C. However, no visitors will see both Experiment B and Experiment C. This is because Experiments B and C are both in the same exclusion group, which makes them mutually exclusive.

One overlapping experiment with two mutually exclusive experiments, some traffic not allocated

In this example, you have three experiments again, and two are mutually exclusive, but some of your traffic is not allocated to either mutually exclusive experiment:

• Experiment A: 100% traffic allocation

• Experiment B: 20% traffic allocation and in Exclusion Group 1

• Experiment C: 50% traffic allocation and in Exclusion Group 1

• Some traffic (30%) in Exclusion Group 1 is not allocated to either Experiment B or C

Again, all visitors will see Experiment A. Experiments B and C are still mutually exclusive, so some visitors will see Experiment B, some will see Experiment C, and none will see both. However, in this case, 30% of visitors will see only Experiment A because 30% of visitors are not allocated to either experiment in the exclusion group.

Experiments in two exclusion groups

You can also use two different exclusion groups to add layers of mutually exclusive experiments. Imagine these experiments and traffic allocations:

• Experiment A: 100% traffic allocation

• Experiment B: 20% traffic allocation and in Exclusion Group 1

• Experiment C: 50% traffic allocation and in Exclusion Group 1

• Some traffic (30%) in Exclusion Group 1 is not allocated to either Experiment B or C

• Experiment D: 50% traffic allocation and in Exclusion Group 2

• Experiment E: 50% traffic allocation and in Exclusion Group 2

In this example, all visitors will see Experiment A, some visitors will see Experiment B, some will see Experiment C, and none will see both B and C. Again, 30% of visitors will not see either Experiment B or C.

But in this case, because of Exclusion Group 2, you have more possibilities. All visitors will see either Experiment D or E, but none will see both. A visitor could see experiments from all three layers: Experiment A, plus one experiment from each of Exclusion Group 1 and Exclusion Group 2. It is also possible for a visitor to see experiments from only two layers, because some traffic is not allocated to either experiment in Exclusion Group 1.

Paused experiments

When you pause an experiment, new visitors will no longer be bucketed into that experiment.

Visitors who have already seen the experiment will remain bucketed into the experiment, although they will not see it again for as long as it is paused. If you resume the experiment, returning visitors who are bucketed into the experiment will begin seeing it again, and new visitors could be bucketed into it.

For example, suppose a visitor is bucketed into Experiment C when they visit your site. If you pause Experiment C, the visitor will not see it anymore. If you resume the experiment, and the visitor returns to your site, they will see Experiment C again because they are still bucketed into it.

Personalization campaigns and exclusion groups

You can also add personalization campaigns to exclusion groups to make experiments and campaigns mutually exclusive. In the example shown below, all visitors will see Experiment A. Some visitors will also see either Experiment B or Campaign A. And some visitors will see only Experiment A.

When you add a Personalization campaign to an exclusion group, the campaign's holdback is not affected.

Two experiments that target different pages

When two mutually exclusive experiments target different pages, the exclusion group decision about which experiment to bucket a visitor in is sticky and deterministic. In other words, if a visitor is bucketed into Experiment A on your site's homepage, and Experiment A is in the same exclusion group as Experiment B on your site's order page, the visitor will not see Experiment B.

Two experiments that target non-mutually exclusive and mutually exclusive audiences

When two mutually exclusive experiments target different audiences that are not mutually exclusive, the exclusion group decision about which experiment to bucket a visitor in is sticky and deterministic. For example, suppose you have two non-mutually exclusive audiences, like "visitors in California" and "all visitors." If a visitor from the "visitors in California" audience is bucketed into Experiment A on your site's homepage, and Experiment A is in the same exclusion group as Experiment B, the visitor will not see Experiment B, even though they also qualify for the the "all visitors" audience.

The same is true when the two mutually exclusive experiments target different pages and non-mutually exclusive audiences, and when you have three mutually exclusive experiments that target the same page.

Two mutually exclusive experiments that target different mutually exclusive audiences, like "Android users" and "iPhone users," are also sticky and deterministic. However, if the audiences for your experiments are mutually exclusive, you don not need to use exclusion groups for the experiments themselves.

Two experiments with different traffic allocation ratios

If an experiment is in an exclusion group, 100% of that experiment's traffic is allocated to the percentage you have allocated to the exclusion group. For example, suppose your Exclusion Group 1 includes two experiments with these traffic allocations:

• Experiment A: 70% traffic allocation

• Experiment B: 30% traffic allocation

In this setup, 100% of the traffic that you send to Experiment A (70% of traffic to Exclusion Group 1) will see Experiment A.

Removing experiments and campaigns from exclusion groups

If you remove an experiment or campaign from an exclusion group, visitors who were previously bucketed into that experiment or campaign are dropped from the exclusion group. This means the visitors will not see any other experiments or campaigns in that exclusion group. If the experiment or campaign is still running and the visitors meet the targeting conditions, they could still see the experiment or campaign.

Duplicating experiments and campaigns in an exclusion group

If you duplicate an experiment or campaign that is part of an exclusion group, the new experiment or campaign will not be associated with any exclusion group.

Changing traffic allocation for experiments or campaigns in an exclusion group

Optimizely Web Experimentation and Optimizely Performance Edge use "sticky" bucketing for the traffic allocation for experiments and campaigns in an exclusion group. This means that if you change the traffic allocation, returning visitors will continue to see the experiments and campaigns they saw before your change. New visitors will see experiments and campaigns according to your new traffic allocation.

For example, suppose you have two experiments in an exclusion group, with these traffic allocations:

• Experiment A: 50% traffic allocation and in Exclusion Group 1

• Experiment B: 50% traffic allocation and in Exclusion Group 1

About half of your visitors will see Experiment A and half will see Experiment B.

If you add two more experiments to the exclusion group and reallocate traffic like this, you have two experiments in an exclusion group, with these traffic allocations:

• Experiment A: 25% traffic allocation and in Exclusion Group 1

• Experiment B: 25% traffic allocation and in Exclusion Group 1

• Experiment C: 25% traffic allocation and in Exclusion Group 1

• Experiment D: 25% traffic allocation and in Exclusion Group 1

With your updated traffic allocation, returning visitors will continue to see the same experiment they saw the first time: Experiment A or B. Returning visitors will not see Experiment C or D. New visitors might see any of the four experiments: about 25% of your visitors will see each of Experiments A, B, C, or D.

Archiving exclusion groups

Before you archive an exclusion group, you must remove all experiments and campaigns in it. Archiving is disabled for exclusion groups with experiments or campaigns in them.

Visitor bucketing IDs

A given visitor will have a consistent bucketing ID each time they return to the same exclusion group, experiment, or campaign to be bucketed. This bucket ID will not be the same for a given visitor across exclusion groups, experiments, or campaigns.

Best practices

For reliable results, you should start all the experiments or campaigns in an exclusion group at the same time, after you added them to the exclusion group. You should not add running experiments or campaigns to an exclusion group because it may have unintended effects on your results.

Debugging tips

Use the JavaScript API and console log for debugging:

?optimizely_log=debug

Look for “Group” in the logs.

You can also check window.optimizely.get('data').groups on a page that has a grouped experiment or campaign running on it. The weightDistributions property will contain a list of entities and their campaign IDs.

If there is at least one experiment running in the exclusion group, all experiments will appear in the Optimizely log, including paused and draft experiments. Paused and draft experiments will not be included in the snippet, nor will they run; however, paused and draft IDs will be present in the exclusion group decision. This means that a visitor could be allocated to a draft experiment, prior to the experiment being published.

If anything less than 100% of your traffic is allocated to an experiment, visitors included in that unallocated percentage will not see any experiment in the exclusion group. If the exclusion group allocation is later increased to include 100% of incoming traffic, visitors who saw no experiment will be evaluated into the experiment that filled the missing percentage.

A value of "none" in the Optimizely log for the group object represents the allocation to the percent of traffic that is not allocated to an experiment. This will only occur when a mutually exclusive group does not allocate 100% of visitors to an experiment.