Multivariate tests for Optimizely Web Experimentation

  • Updated
  • Optimizely Web Experimentation

Multivariate tests (MVT) let you change several elements on your site and determine which combinations of changes perform best.

Creating a MVT is similar to creating a standard A/B test, except that you define multiple sections in a MVT. You can think of each section as a self-contained A/B test that modifies a single element, like a button or image. By creating multiple sections, you can test these elements at once.

You can create multiple variations within each section. Optimizely Web Experimentation automatically generates all the combinations of these sections as you build variations. Each combination pulls a variation from each of the sections you have already defined and rolls them together into a composite experience for your visitor. The combinations included in your MVT are visible in the Combinations tab.

The combination names are always made up of a series of capital letters. Each of these letters represents a variation from a particular section. For example, an MVT with three sections would have a combination named ABA. This combination would consist of the first variation of the first section (A), the second variation of the second section (B), and the first variation of the last section (A).

Below each combination name, you can also find a clear description of the different variations included in it. In the former image, the combination labeled ABB is testing a combination of the original variation's button text color, a new button background color, and a different button copy for the button's text.  

Traffic allocation in MVTs: Full factorial and partial factorial

Optimizely Experimentation offers two different approaches to allocating traffic in multivariate experiments: full factorial or partial factorial.

In the full factorial approach, Optimizely Experimentation tests every combination of variations against all the other potential combinations, allocating traffic on the basis of section variations. However, this affects the amount of traffic delivered to each combination.

Optimizely multiplies each of the variations' percentages together to determine the percentage of traffic allocated to a particular combination. For example, in a two-variation combination, if Variation 1 has a traffic allocation of 50% in section 1 and Variation 2 has an allocation of 25%, the total traffic allocation for that specific combination will be 12.5% (.50 x .25 = .125).

The advantage of the full factorial approach is that it is more rigorous. However, one drawback is that it requires higher levels of traffic to generate a significant result. Another is that you might end up testing specific combinations that may not make sense to test at all: for example, you would probably expect that testing a blue button against a blue background might not be a successful experiment.

In cases like this, you can use a partial factorial approach. Partial factorial lets you allocate traffic at the combination level, so you can completely exclude specific combinations from your experiment simply by setting their traffic allocations to zero. (By contrast, full factorial traffic allocation happens at the section level, making it impossible to prevent specific combinations from appearing.)

You can also set up your own test layouts, like Taguchi templates that mix variations in specific ways that ensure equal representation, but do not require all combinations to actually be generated. The advantage of partial factorial is that you can test far fewer combinations than full factorial while still enjoying many of the same benefits.

Create a new multivariate test in Optimizely Web Experimentation

To create a new multivariate test:

  1. Go to Experiments > Overview.

  2. Expand Create New and select Multivariate Test.

  3. The New Multivariate Test window displays.
    mceclip1.png

  4. In the Name field, enter a name for your experiment.

  5. In the Description field, optionally enter a description for this experiment. 

  6. Set up targeting to define where the experiment runs. Expand the Target By dropdown list and select either URL or Saved Pages:

      • If you choose URL, enter the address of the target page in the text box.

      • If you choose Saved Pages, select the target page from the list of pages.

  7. In the Sections area, add as many variations as you need.

    The number of possible combinations in an MVT is capped at 64. This works out to, at most, six sections with two variations each (2^6 or 2x2x2x2x2x2). You should also remember that as the number of combinations in an MVT increases, the amount of traffic going to each one decreases, and results take longer to achieve.

  8. Adjust the Traffic Allocation as needed.

  9. Click Create Experiment to save your changes.

Edit variations

Once your experiment is set up, you can access your variations from either the Sections or Combinations tabs. The Sections tab displays a complete list of the sections you created for your experiment and all the variations contained within each one. Click on the name of the variation or Edit to view and update the variation within the visual editor.

If you prefer to view the outputted combinations themselves, you can do so from within the Combinations tab. Click on Preview next to the name of the combination, or click on the combination name you want to view.

Results

Results for an MVT work like a typical A/B experiment, with data gathered for each of the combinations.

mceclip2.png

If you are working with a large number of sections or variations in your MVT, each combination will likely take longer to reach significance due to the lower traffic volumes each one receives. With section rollups, you can collapse an MVT into a single A/B test on a single section. This way, you can get a sense of effect sizes and significance along that section, even when traffic is spread thin. See Section rollups in multivariate tests.

Multivariate testing FAQs