Guide your program with an experimentation charter

Once you have Optimizely Experimentation ready to start running experiments, consider how you can make the most effective use of the platform. If you immediately jump into testing, you may face:

• Inefficient test planning and set up of missed opportunities

• Tests launched with errors or mistakes, or missing goals or audiences

• Low velocity

• Poor communication between team members

• Tests that generally have no impact

• An inability to effectively learn from test results

Plan before you start testing to avoid these scenarios. A charter helps you track the mission, resources, and workflow that govern your A/B testing strategy.

Materials to prepare

Address the following questions before creating your testing charter.

• Business requirements – What are the core responsibilities of the optimization team?
• Budget – What resources do you have?
• Performance benchmark data – What time investments are required to meet performance goals?
• Executive sponsorship – Do you have approval for resources, including time investments?
• Team proficiency – What experience and skill sets do available team members have?

People and resources

• Executive sponsor
• Program manager
• Team leads (technical, design, analytics)
• External consultant, if needed

Actions you will perform 

• Create a testing charter.
• Map a workflow.
• Budget time and resources.
• Benchmark the team's proficiencies.
• Evaluate whether to work with a Solutions Partner.
• Secure approval from stakeholders.

Deliverables

• A testing charter that documents roles and responsibilities, including:
  • Role definitions
  • Time (in percent) spent per role
  • Program and individual metrics
• Optimization workflow
  • Inputs and outputs for each step of the workflow
  • Responsibilities and contributions by step
• Evaluation of team's skill sets against program initiatives and goals

Watch out for:

• Difficulty establishing roles and responsibilities before creating a well-articulated workflow.
• Challenges to establish staffing requirements and time investments before you have a prioritized roadmap of tests and campaigns.

What to include

Your testing charter may include a brief mission statement but should include details to help anyone new or unfamiliar with the testing program learn more. You do not need to include everything from the list below, but you should be thorough for your team.

The testing charter should be a working document, owned by the program manager (or equivalent). One or two months into the testing program, the program manager should revisit the document to see how the actual process compares to this plan and make any appropriate modifications. The program manager can then review the charter on a quarterly or semi-annual basis.

To get started, download the testing charter template (.rtf). You can copy and modify this template in the program of your choice. Here are the explanations for each section of the document:

• Introduction – Set the context and define the purpose for the rest of the charter. Why did you purchase the Optimizely Experimentation testing platform? What problems were you trying to solve?  What are the long-term goals and expected outcomes?

• Team – Define the testing team and include a description of their roles defined by the Optimizely Experimentation platform privileges.

• Process – Create a workflow diagram in Keynote or PowerPoint from conception, execution, and results-sharing to (potentially) onsite implementation. Add a brief description and a list of key participants for each of the steps you outline.

• Meetings and meeting cadence  – Decide when to have brainstorming and status meetings to check on upcoming tests and review ongoing or completed ones. Decide how often the testing team meets with a broader group.

• Test Velocity – Set a goal for testing velocity and modify it as needed based on the actual testing pace at your company. You may want one test always running, ten tests per week, or twenty-five per quarter.

• Test plan and results templates – Create a link to the most updated testing template the company uses.

• Links key – Link to all other essential information, including your current testing roadmap, templates (described above), and an archive of test results. If team members bookmark this page, they should be able to find any other relevant information they need. You may also want to link to pages that document definitions of the following:

  • Test goals – For example, Purchase Confirmation – a pageview goal targeted to http://www.example.com/thank_you_page.

  • Targeting conditions – For example, Product Pages – targeted with a substring match to example.com/productID.

  • Audiences and dimensions – For example, Category Affinity defines users who are interested in a particular category and bases it on a recent product or category pageview associated with that category.

• Other guidelines  – Decide what guidelines and rules your team will follow when it comes to testing. Do different types of tests have different processes for QA and approval? Are there standard goals included in every test? When will a winning variation be served through the Optimizely Experimentation platform, and when will it be hard coded to the site?