Create a basic experiment plan

  • Updated
This topic describes how to:
  • Create a basic experiment plan for a single test
  • Communicate your experiment design to external stakeholders

So, now you have a prioritized list of ideas for optimization. Great! Your next step is to implement the top ideas in that list. For many teams, the challenge at this stage is planning the scope, design, and implementation of a test with multiple stakeholders.

A basic experiment plan can help you scope and launch individual experiments. You will estimate the cost of a test in terms of traffic and time—and weigh that cost against the potential value it will yield.

To turn a line item on your roadmap into an experiment or campaign, create a single deliverable that covers the 5Ws + How:

  • Why are you running this experiment (hypothesis)?

  • When and where will your variations run?

  • Who do you want see this experiment?

  • What changes does your variation make?

  • How are you measuring success?

This article helps you create a basic experiment plan. Use it to design and implement individual ideas and communicate with stakeholders. Share it with the strategists, designers, developers, and approvers involved in your optimization efforts to secure approval and improve visibility.

Materials to prepare
  • Experiment hypothesis
  • Business goals
  • Variation descriptions (wireframes or screenshots)
  • Summary of all technical and design assets needed for the experiment
  • Parameters for significance and lift that indicate the change will be implemented permanently

People and resources

Actions you will perform 

  • Create a test plan document
  • Create a rigorous QA checklist
  • Review and update plan with stakeholders
  • Confirm scope of test
  • Define primary, secondary, and monitoring goals
  • Confirm stakeholders who will create required resources
  • Document responsibilities and deadlines (in Kanban, gantt chart, or other internal method)
  • Finalize test plan
Deliverables
  • Test plan document containing:
    • All details for building an experiment
    • Technical requirements
    • Scope of the experiment
    • Creative assets or wireframes
    • Screenshots of variations
Watch out for:
  • Ill-defined scope
  • Lack of true hypothesis or goals
  • Lack of executive buy-in
  • Missing screenshots
  • Poor understanding of resource needs
  • Inaccurate effort estimates
  • Inadequate documentation for QA
  • Plan not shared with the proper stakeholders
  • Lack of adherence to experiment plan when building the test
If your experimentation program is more mature, check out our advanced experiment design template and QA checklist instead.

Create a basic experiment plan

Download this template to create your own experiment plan.

This Test Idea Worksheet below can also help you visualize your experiment:

For more downloadable resources, check out the Optimizely Experimentation Testing Toolkit.

Use this basic plan to manage the project and set expectations for external stakeholders. The 5Ws + How to establish the fundamental intentions of the proposed test for developers and designers who help you execute the plan. Provide all details needed to build and QA the experiment.

The plan also standardizes your testing practice and streamlines the approval process with stakeholders outside of your team. Create a list of all QA visitor use cases or all the ways that a visitor arrives or navigates through the experiment along with the expected result. Your QA team will use this list to evaluate the experiment before launch.

Create this plan as a shareable document that multiple stakeholders can reference: a presentation slide, an email template, or a wiki page that covers all basic information about the test. Strategic planners use this document to communicate the concept to the designers and developers responsible for implementing the experiment.

For maximum visibility, link your individual test plans to your prioritized list of ideas.

Define primary, secondary, and monitoring goals

When you create your experiment plan, decide how you will measure success. In Optimizely Experimentation, the primary goal that measures how your changes affect your visitors' behaviors. Consider setting secondary and monitoring goals as well to gain greater visibility into your customers' behaviors and make sure the lift you see sets your program up for long-term success. 

To learn more, read this article on primary, secondary, and monitoring goals.

Decide what type of experiment to run

The type of experiment depends on how you expect your changes to impact your primary conversion event.

The minimum detectable effect (MDE) can help you decide what type of test to run.

A/B test

Run an A/B test when improvement in your primary goal can be attributed to a single change in your user experience.

Multivariate test

If it is important for your business to precisely measure how multiple changes are interacting with each other and influencing conversions, create a multivariate test that evaluates each combination of variables against all others.

Multi-page test

If you are measuring success in conversions across a series of pages, a multi-page test helps you measure how changes affect how visitors through each stage of the funnel.

A/B/n test

If you want to test multiple changes but do not want to run a full multivariate test—which can be costly in terms of time and traffic—consider a blended A/B/n approach.

Test multiple versions of your page (A, B, and n more pages) without comparing all possible combinations of variations against each other. This test type is more economical, while still allowing you to attribute lift to certain changes.