Experiment Types: AB, Multivariate, and Multi-page

Optimizely Web Experimentation and Optimizely Performance Edge provide three different experiment types: A/B testing, multivariate testing, and multi-page (funnel) testing. Learn their differences and evaluate the benefits and drawbacks of using each type.

See:

• How to create an A/B test.

• How to create a multivariate experiment in Optimizely Web Experimentation.

• How to create a multi-page (funnel) experiment in Optimizely Web Experimentation.

Differences among campaigns, experiments, experiences, and variations

Campaigns, experiments, experiences, and variations are terms related to Optimizely Experimentation testing. In general, campaigns have experiences, and experiments have variations. 

• Campaigns – The framework for organizing your strategy in Optimizely Personalization. A campaign takes certain content from your site, like the promotions on your homepage, and exchanges it for different content for different audiences. See Personalization campaigns.
• Experiment – Optimizely Experimentation's term for an A/B, multivariate, or multi-page test. You might also see experiments called Tests or Campaigns. 
• Experiences – The different personalized versions of your site that you create for visitors based on the page they visit and the audiences they qualify for. Learn more about Personalization experiences. When you are unsure what type of experience works best for a particular audience, you can test experiences with Experimentation in Personalization campaigns.
• Variations – The alternate versions of your site that you test against your original (or baseline) in Optimizely Web Experimentation. Learn how to create and change variations in your experiments.

A/B testing 

A/B testing, also known as split testing, is a method of website optimization in which the conversion rates of two versions of a page—version A and version B—are compared to one another using live traffic. Site visitors are bucketed into one version or the other.

By tracking the way visitors interact with the page—the videos they watch, the buttons they click, or whether or not they sign up for a newsletter—you can determine which version of the page is most effective.

Common uses

A/B Testing is the least complex method of evaluating a page design and is helpful in various situations.

One of the most common ways of A/B testing is to test two different design directions and compare them with each other. 

Example:

The current version of a company's home page might have in-text calls to action, while the new version might eliminate most text but include a new top bar advertising the latest product. After sufficient visitors are funneled to both pages, the number of clicks on each version of the call to action can be compared.

Even though many design elements are changed in this kind of A/B test, only the impact of the design on each page's business goal is tracked, not individual components.

A/B testing can also be used as an optimization option for pages where the effectiveness of only one design element needs to be evaluated.

Example:

A pet store that runs an A/B test on their site might find that 85% more users sign up for a newsletter when it is displayed as held up by a cartoon mouse than one that emerges from a boa constrictor's coils. In A/B testing, a third or fourth version of the page can be included in the test, which is called an A/B/C/D (or A/B...n) test. This means that traffic to the site must be split into thirds or fourths, with a lower percentage of visitors visiting each site.

Advantages

Simple in concept and design, A/B testing is powerful and widely used.

Using a small number of tracked variables lets these tests deliver reliable data, as they do not need much traffic. This is helpful if your site has a small number of daily visitors.

A/B testing is easy to interpret and some large sites can use it as their primary testing method to run cycles of tests one after another instead of more complex multivariate tests.

Disadvantages

A/B testing is best used to measure the impact of two to four variables on interactions with the page. Tests with more variables take longer to reach statistical significance, and A/B testing does not reveal any information about the interaction between variables on a single page.

Multivariate testing (MVT)

Multivariate testing uses the same core mechanism as A/B testing but compares a higher number of variables and reveals information about how they interact with one another. Think of it as multiple A/B tests layered on top of each other.

As in an A/B test, traffic to a page is split between different design versions. Then, a multivariate test measures the effectiveness each design combination has on the business goal.

When a site receives enough traffic to run the test, the data from each variation is compared to find the most successful design and reveal which elements have the most significant positive or negative impact on a visitor's interaction.

Common Uses

Multivariate testing is most commonly done on pages that need several elements evaluated. For example, a page includes a sign-up form, header text, and a footer.

To run a multivariate test on this page, instead of creating a significantly different design as in A/B testing, you can create two different lengths of the sign-up form, three different headlines, and two footers. Next, you can funnel visitors to all possible combinations of these elements.

Testing all possible combinations of a multivariate test is also known as full factorial testing. It is one of the reasons why multivariate testing is recommended only for sites with a significant amount of daily traffic. The more variations that need to be tested, the longer it takes to obtain meaningful data from the test. It is, however, the most accurate way to run a multivariate test.

For example, varying a page footer may have minimal effect on the page's performance, while running the length of the sign-up form can have a huge impact.

Advantages

Multivariate testing lets you target redesign efforts to the elements of your page where they have the most impact. This is useful for designing landing page campaigns. For example, the data on the effect of an element's design can be applied to future campaigns, even if the context of the element context changes.

Limitations

The most significant limitation of multivariate testing is the amount of traffic needed to complete the test. Because experiments are fully factorial, multiple changing elements can result in many possible combinations that must be tested. Even a site with reasonably high traffic can have trouble completing a test with more than 25 combinations.

When using multivariate tests, consider how they fit into your testing cycle and redesign. Even if you have information about the impact of a particular element, you may want to do additional A/B testing cycles to explore other ideas. In some cases, several well-designed A/B tests may be preferable over the extra time needed to run a full multivariate test.

Multi-page testing

Multi-page (also known as "funnel") testing is similar to A/B Testing, but instead of making variations to a single page, the changes are implemented consistently over several pages. Like A/B testing, site visitors of a multi-page test are bucketed into one version. You can determine which design style is most effective by tracking how these visitors interact with the different pages they see. The key to getting usable data in a multi-page test is preventing users from seeing a mix of variations; they should see a consistent variation throughout a set of pages. This lets one variation be well-tested against another.
 

Common Uses

Testing different design directions with each other can be done quickly using multi-page testing. For example, imagine an ecommerce website that lets users search through numerous products, add desired items to a virtual shopping cart, and then purchase the items.  

In this case, users see more than a single page. Instead, they are funneled through several pages before purchasing or leaving the website. Using a multi-page test, you can create two (or more) unique designs for a set of pages. After you do this, ensure that your users see only one design style throughout all the pages.

After enough visitors are funneled through the different designs, the effect of the other design styles can be compared effectively.

Advantages

Like A/B testing, multi-page testing is simple and can provide meaningful and reliable data quickly and easily. It lets all users see a consistent set of pages, whether it is the original or a redesigned variation. 

Multi-page testing lets you implement the same changes you make on a single page in a typical A/B test to several pages. This ensures your visitors do not bounce between different variations and designs when funneling through your website.

Limitations

Multi-page testing has many of the same limitations as A/B testing. Like A/B testing, multi-page testing is best used to measure the impact of only a few variables at a time. Tests with too many variables take longer to run. Determining the effect of each change you make on every page is also more challenging.

When you set up a multi-page test, you must have the same number of variations for every page of the experiment. An uneven number of variations creates inconsistency between pages, worsens the user experience, and makes any data collected challenging to interpret. Additionally, only targeting conditions that apply to all pages in the experiment can be used for multi-page experiments.