Design an effective hypothesis

Strong hypothesis statements help you turn data and insights about visitor behavior into focused proposals for action. Each hypothesis is an idea to be tested; every idea confirmed or rejected informs you about visitor expectations and behaviors to improve optimization. Consistent hypothesis-driven experimentation helps your program make an impact on company goals. Learn more with the Optimizely Academy courses.

For example, you notice a high abandonment rate in your purchase funnel. You hypothesize that links in the funnel distract visitors, so you experiment with removing them. The improvement in completed purchases is about more than just a lift. It confirms your hypothesis and understanding of your visitors and their site experience. Use this insight to help decide what to optimize next, why, and how to measure results.

Programs that do not use hypotheses risk wasting resources on unfocused experimentation that fails to make a business impact.

Download the hypothesis worksheet (.pdf) to start.

What is a hypothesis?

A hypothesis is a prediction you create before running an experiment. The common format is: 
If [CAUSE], then [EFFECT], because [RATIONALE].

Strong hypotheses consist of three distinct parts: a definition of the problem, a proposed solution, and a result.

Problem

Solution

Propose a solution. Describe the change so someone who reads the hypothesis can understand the change without screenshots. Add a rationale that explains why this solution is the right one to solve the identified problem.

Proposed solution – Move the filters to the left side of the results.

Rationale – This is the most common filter location; users are more likely to notice them there.

Result

Predict a result that connects your hypothesis with key business metrics. Include metrics that determine the success or failure of your experiment. Decide on the specific metrics to track success in the experiment or campaign.

Primary metric – % of users who use filters increases.

What success looks like – % of users who move on to a product page increases; the purchase rate increases.

Examples of strong hypotheses

Example 1

• Problem – Users find the featured products on the homepage irrelevant. Only 12% of users click on them, and 9 out of 11 say they never found interesting products on the homepage.
• Solution – Set the algorithm for featured products to display products from recent categories the user has visited. If users visit a category, they have expressed interest in those products.
• Result – The percentage of users who click on the featured products and the percentage of users who added a product to the cart increases.
• What makes this strong? 
  • Qualitative and quantitative validations of the problem

  • The solution is based on a common UI practice in the industry 

Example 2 

• Problem – Users do not understand the name on the financial services tab, as explained in usability interviews, and 85% of users drop off after getting to the page.
• Solution – Try different names for the sections, such as: “financial services,” “professional services,” or “money."
• Result – More clicks on the tab and a smaller drop-off rate on the section landing page.
• What makes this strong?
  • Clear problem identification, including symptom 
  • The results encapsulate quantity (more clicks to section) and quality (less drop off at section)

Best practices

Engage in discovery

The difference between a well-formulated hypothesis and a guess is data. Build a business intelligence report. Carefully observe your customer’s journey through your site with direct data sources like web analytics data and indirect data sources like competitor overviews.

Use data linked to your company’s goals to ensure you focus on areas of impact rather than making UX changes in isolation. Data sources can help you validate the problem and the solution rationale. Avoid generating hypotheses based on intuition.

Create a testable hypothesis

To efficiently test your hypothesis, identify the metrics to track and define clear criteria for success and failure. 

For example, you hypothesize that removing breadcrumb navigation from the checkout page will help visitors stay in the funnel and increase conversions. The difference between the original and the variation is the presence or absence of breadcrumbs. The effect of that change can be measured in the number of conversions.

Use insights as a learning opportunity

Hypothesis-driven experimentation gives you insight into visitor behaviors. These insights generate additional questions about your visitors and site experience and drive an iterative learning process.

The cycle should follow this general pattern:

1. Gather data about your visitors' behaviors and industry, and use insights from that data to ask questions.

2. Formulate a hypothesis based on insights from your data.

3. Design and implement an experiment or campaign based on your hypothesis.

4. Analyze your results to decide whether your hypothesis is confirmed or rejected.

5. Create and document conclusions.

6. Use conclusions to create questions.

Connect to your company’s problems

Begin your process with a problem, not a solution. If a solution fails to deliver the expected result, use the problem statement to explore potential solutions and iterate. This practice lets you focus on the problems that prevent your company from reaching its goals.

Learn about connecting your optimization program to company goals.