Create an advanced experiment plan and QA checklist

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Personalization
  • Optimizely Performance Edge
  • Optimizely Feature Experimentation
  • Optimizely Full Stack (Legacy)
This article is part of The Optimization Methodology series.

A full experiment plan gathers decisions from stakeholders into a single, collaborative document. It provides a detailed summary of the motivations and mechanics of your experiment or campaign. When shared, it helps build visibility across the organization and scale your experimentation strategy by establishing a standardized process.

A QA checklist builds efficiency around your QA process to ensure your experiments and campaigns work as intended.

Materials to prepare
  • Experiment hypothesis
  • Business goals
  • Variation descriptions (wireframes or screenshots)
  • Summary of all technical and design assets needed for the experiment
  • Parameters for significance and lift that indicate that the change will be implemented permanently
People and resources Actions to perform 
  • Create a test plan document.
  • Create a rigorous QA checklist.
  • Review and update plan with stakeholders.
  • Confirm scope of test.
  • Define primary, secondary, and monitoring goals.
  • Confirm stakeholders who will create required resources.
  • Document responsibilities and deadlines (in Kanban, Gantt chart, or another internal method).
  • Finalize test plan.
Deliverables
  • Test plan document containing:
    • Details for building an experiment
    • Technical requirements
    • Scope of the experiment
    • Creative assets or wireframes
    • Screenshots of variations
  • QA checklist (.xlsx)
Watch out for:
  • Ill-defined scope
  • Lack of true hypothesis or goals
  • Lack of executive buy-in
  • Missing screenshots
  • Poor understanding of resource needs
  • Inaccurate effort estimates
  • Inadequate documentation for QA
  • Plan not being shared with the proper stakeholders
  • Lack of adherence to plan when building the test

If your team is starting to run its first tests, see the basic experiment plan.

Experiment design document template

Your experiment design document should include:

  • The actual code used for implementation

  • Alignment with sprint planning

  • Experiment ID

  • Roles/responsibilities

  • Primary, secondary, and monitoring goals for the experiment

  • Correlating goals to business value

You may also want to use minimum detectable effect to choose what type of test to run.

QA checklist template

When you plan an experiment, include the QA team so they can create a checklist. This helps build a rigorous, efficient process for cases that need to be checked and eliminates those that do not.

Your QA checklist should include:

  • Goals or events in the experiment and how each is triggered

  • Added functionality

  • Common use cases including expected user flows to and from the page

  • Included or excluded audiences

  • URLs where the experiment should run

  • Sample workflow to fire a goal (especially for custom events)

Your QA team should grade each use case as pass or fail. Push the experiment live once everything on the list has passed the test.

QA for separate development and production environments

If you can access separate development and production environments, notify the development team before running the experiment on the site. Maintaining separate environments reduces the risk of deploying an unfinished experiment to your live site. Build experiments and QA with the following process:

  1. Ensure each environment has Optimizely Experimentation implemented.

  2. Build your experiment in the development environment.

  3. QA the development environment.

  4. Push the experiment live in the QA environment. Confirm goal firing on the Results page and all analytics data collection.

  5. Duplicate the experiment into your production environment.

  6. Set a test cookie so only you can see the experiment on the live site.

  7. Push the experiment live in the production environment and QA.