Mutually exclusive experiments in Feature Experimentation

In Optimizely Feature Experimentation, experiments may overlap, exposing a single user to multiple experiments simultaneously. Mutually exclusive experiments prevent this by ensuring a user is exposed to only one particular experiment at a time. For experiments that are not mutually exclusive, Feature Experimentation uses a unique, random value for each experiment to determine user entry. This value is not mutually exclusive across experiments, which means users may enter multiple experiments. For more information, see How bucketing works.

Example

For example, there are two experiments: Experiment A and Experiment B. Each experiment receives 20% traffic allocation (the percentage of total traffic that is eligible for the experiment). The traffic allocation would be the following:

• 16% of traffic falls in Experiment A only.
• 16% of traffic falls in Experiment B only.
• 4% of traffic falls in both Experiment A and Experiment B.
• 64% of traffic is not in any experiment.

In the example above, Experiment A's and Experiment B's results may be skewed. If users who see both A and B behave differently from those who see just A or B, then the results for A and B are skewed by the overlap. This is called an interaction effect.

If Experiment A and Experiment B are mutually exclusive, Feature Experimentation chooses the same random value (unique to the exclusion group) to bucket users in experiments A and B. This method ensures that experiments cannot overlap for the same users. If experiments A and B are mutually exclusive, the traffic allocation looks something like the following:

• 20% of traffic falls in Experiment A only.
• 20% of traffic falls in Experiment B only.
• 60% of traffic is not in any experiment.

Feature Experimentation also ensures mutual exclusivity between experiments in an exclusion group that run at different times.

Best practices

Before implementing exclusion groups and mutual exclusion, keep the following best practices in mind:

• Make experiments mutually exclusive only when required to prevent restricting traffic more than necessary.
• Experiments within an exclusion group do not need to start or stop at the same time, but they must all be part of the exclusion group from the moment the first experiment starts until the last experiment finishes to maintain a fixed traffic allocation and visitor bucketing.
• No experiments should be added to the group after one of them has been started. If experiments are added to the group while any of the group's existing experiments are already running, traffic allocation to the experiments will be shifted, and visitors may be exposed to multiple experiments after all.
  • For example, a visitor is initially bucketed into Experiment A and has already been exposed to it. Then, Experiment B gets added to the exclusion group. Traffic allocation shifts, and this visitor may now be assigned to Experiment B while they no longer seeing Experiment A.
• Do not remove running experiments from an exclusion group or reallocate traffic in ways that could introduce bias or overlap between experiments.
  • For example, an exclusion group contains three experiments. A visitor is initially bucketed into Experiment B. Then, Experiment C is removed from the group, leaving only experiments A and B. With the bucketing ranges now being shifted, the visitor may get bucketed from Experiment B into Experiment A.

Exclusion group implementation

In Feature Experimentation projects, you can make two or more experiments mutually exclusive of each other using exclusion groups. Using exclusion groups does not require any change in the SDKs. The Decide and Track Events methods automatically evaluate whether an experiment is in a group and assign users to the appropriate experiment and variation.

If you want to implement exclusion groups using the Feature Experimentation APIs instead, see Use mutual exclusion in the developer documentation.

Create an exclusion group

To create an exclusion group in the UI, complete the following:

1. On the Flags dashboard, select the Exclusion Groups tab.

2. Click Create New Exclusion Group. 

   

3. Name your exclusion group.
4. (Optional) Add a Description.
5. Select an Environment.
6. (Optional) Select an experiment to add your exclusion group.

7. Click Create Exclusion Group. 

   

Add an experiment to an exclusion group

To add an experiment to an exclusion group before running your experiments, complete the following:

1. Click the flag that contains your experiment rule.
2. Select your environment and experiment rule.
3. In the Configure Rule section, select the exclusion group from the drop-down menu under Add this experiment to the following exclusion group.
4. Add the percentage of the exclusion group traffic that should be allocated to the experiment.

5. Click Save Experiment. 

   

Remove an experiment from an exclusion group

Remove finished experiments from the exclusion group to make space for other experiments.

1. Click the flag that contains your experiment.
2. Select your environment and experiment.
3. In the Configure Rule section, select None from the drop-down menu under Add this experiment to the following exclusion group.
4. Click Save Experiment.

Archive and unarchive an exclusion group

Archive

To archive an exclusion group:

1. On the Flags dashboard, select the Exclusion Groups tab.
2. Click the Actions icon (...) of the exclusion group you want to archive.

3. Select Archive. 

   

Unarchive

To find archived exclusion groups and unarchive them:

1. On the Flags dashboard, select the Exclusion Groups tab.
2. Filter the Status drop-down list to show Archived exclusion groups.
3. Click the Actions icon (...) of the exclusion group you want to unarchive.

4. Select Unarchive. 

   

Bucketing

The following table shows how exclusion groups can interact with other bucketing methods.