Guide your program with an experimentation charter

  • Updated
This article is part of The Optimization Methodology series.
This topic describes how to:
  • Understand the purpose of a testing charter
  • Decide what is important to include in your charter
  • Build your own testing plan using our starter template

Imagine the following scenario: You recently purchased a new A/B testing platform for your company. You have high hopes of showing a great investment return after having spent weeks (or maybe months) deciding on the platform, negotiating the contract’s duration and cost, and hiring a new employee (or transitioning a whole team) to use it. The platform’s code has been installed, and it is now possible to begin testing.

The question is, how ready are you? From the initial consideration phase to purchase and installation, it is uncommon that the company has fully considered the necessary details of how they will make the most effective use of the platform. But if you immediately jump into testing, in the weeks and months that follow, you may find yourself facing:

  • Inefficient test planning and setup of missed opportunities

  • Tests launched with errors or mistakes, or missing goals or audiences

  • Low velocity

  • Poor communication between team members

  • Tests that generally have no impact

  • An inability to effectively learn from test results

These scenarios can be avoided or attenuated with a little planning before you start. Broadly speaking, documenting a game plan allows for efficient, consistent, and meaningful testing. A charter helps you track the mission, resources, and workflow that govern your A/B testing strategy.

Materials to prepare

Address the following questions before creating your testing charter.
  • Business requirements: what are the core responsibilities of the optimization team?
  • Budget: what resources do you have?
  • Performance benchmark data: what time investments are required to meet performance goals?
  • Executive sponsorship: do you have approval for resources, including time investments?
  • Team proficiency: what experience and skill sets do available team members have?
People and resources
  • Executive sponsor
  • Program manager
  • Team leads (technical, design, analytics)
  • External consultant, if needed
Actions you will perform 
Deliverables
  • Testing charter that documents roles and responsibilities, including:
    • Role definitions
    • % time spent per role
    • Program and individual metrics
  • Optimization workflow
    • Inputs and outputs for each step of the workflow
    • Responsibilities and contributions, by step
  • Evaluation of team's skill sets against program initiatives and goals
  • Establish Optimizely Experimentation user roles
Watch out for:
  • Difficulty establishing roles and responsibilities before creating a well-articulated workflow
  • Challenges to establish staffing requirements and time investments before you have a prioritized roadmap of tests and campaigns

This article is a part of the Optimization Methodology series.

What to include

Your testing charter (which you might document in a company wiki, a Word document, a Google Doc, etc.) may include a brief mission statement, but should definitely include many practical, referenceable details. When you create it, imagine a new employee, senior company leader, or anyone at your company who does not know about the testing program. For him or her, the testing charter is a great place to start learning about the program. Your document does not need to include everything from the list below, but the more thorough you are based on the needs of your team, the better off you will be.

The testing charter should be a working document. It will never be finished. The testing Program Manager (or equivalent) should own the charter. One or two months into the testing program, the Program Manager should revisit the document to see how the actual process compares to this plan. Depending on what is working or not, this is a good time to modify the plan. From there, the Program Manager can ensure that the program is meeting its goals by reviewing and updating the charter on a quarterly or semi-annual basis.

To get started, download our testing charter template (.rtf). You can copy and modify this template in the program of your choice. Below you can find more detailed explanations for each section of the document.

  • Introduction: Set the context with a paragraph of the introduction. Why did you purchase the Optimizely Experimentation testing platform? What problems were you trying to solve?  What are the long-term goals and expected outcomes? Define the purpose for the rest of the document.

  • Process: What is the workflow from conception, execution, and results-sharing, through to (potentially) onsite implementation? Create a workflow diagram in Keynote or PowerPoint and insert the image here.

    For each of the steps you outline, add a brief description and a list of key participants in each step.

  • Meetings & Meeting Cadence: When do you have testing brainstorm meetings? When do you have status meetings to check on upcoming tests and review currently running and recently completed ones? How often does the testing team meet with a broader group?

  • Test Velocity: What is the goal for testing velocity: One test always running, ten tests per week, twenty-five per quarter? Whatever it is, set an initial goal and modify this as needed to ensure that it is reasonable based on the actual testing pace at your company.

  • Test Plan & Results Templates: Testing programs that are efficient run meaningful tests, act on the lessons of each test, and create testing documentation. Create a link to the most-updated testing template the company uses to document test plans and results.

  • Links Key: Your links key is where you link to all the other essential information, including your current testing roadmap, templates (described above), and an archive of test results. If team members bookmark this page, they should easily be able to find any other relevant information they need. You may also want to link to pages that document definitions of the following:

    • Test Goals. For example, “Purchase Confirmation – a pageview goal targeted to http://www.example.com/thank_you_page.”

    • Targeting Conditions. For example, “Product Pages – targeted with a substring match to example.com/productID.”

    • Audiences and Dimensions. For example, “Category Affinity defines users who are interested in a particular category and bases it off of a recent product or category pageview associated with that category.”

  • Other Guidelines & FAQs: What guidelines and rules will your team follow when it comes to testing? Do different types of tests have different processes for QA and approval? Are there standard goals included in every test? When will a winning variation be served through the Optimizely Experimentation platform, and when will it be hard coded to the site?