Design an effective hypothesis

  • Updated
This topic describes how to:
  • Form your own data-driven hypothesis
  • Create strong hypotheses using "Problem, Solution, Result" framework
  • Identify the components of an effective hypothesis

A strong hypothesis is the heart of data-driven optimization. Hypothesis statements help you turn a wealth of data and insights about your visitors’ behavior into focused proposals on which you will take action.

Use hypotheses to drive meaningful experimentation. Each hypothesis is an idea to be tested; every idea that is confirmed or rejected informs you about your visitors’ expectations and behaviors, and feeds your iterative optimization process. Consistent hypothesis-driven experimentation helps your program make an impact on company goals. Learn more in our Optimizely Academy course, Write an Effective Hypothesis.

If you are unsure of the extent of your organization's experimentation capabilities, Optimizely's Maturity Model can help you determine what kind of experiments you should be doing, as well as the next steps your company should take to maximize your returns from experimentation.

For example, imagine that you notice a high rate of abandonment in your purchase funnel. You hypothesize that links in the funnel are distracting to visitors, so you experiment with removing them. The improvement you see in completed purchases is about more than just lift. It confirms your hypothesis and understanding of your visitors and their site experience. Use this insight to help decide what to optimize next, why, and how you will measure results.

Hypothesis-driven optimization sets your company up for long-term gains:

  • Build a mechanism for constant inquiry 

  • Understand your users' pain points

  • Discover and prioritize the potential benefits of your experiments

  • Establish a common language for ideation and research

  • Connect experiments to your company’s challenges

Programs that do not use hypotheses risk wasting resources on unfocused experimentation that fails to make a business impact.

Download this hypothesis worksheet (.pdf) to start.

What is a hypothesis?

A hypothesis is a prediction you create prior to running an experiment. The common format is: 
If [CAUSE], then [EFFECT], because [RATIONALE].

In the world of experience optimization, strong hypotheses consist of three distinct parts: a definition of the problem, a proposed solution, and a result.

Each component is discussed in greater detail below.

Problem

Experience optimization is most impactful when it solves a problem in the customer experience. When defining a hypothesis, start with a meaningful problem: an issue or pain point in your visitor’s experience that you would like to solve.

Use qualitative and quantitative sources to validate your problem. Often, it can be tempting to slide into assumption when thinking about the user experience. It is important to use data to confirm the issue you are trying to solve.

Explain the problem from the visitor's perspective. Doing so will help you understand the issue more deeply and generate hypotheses that get to the root cause.

For the problem validation, try the following format:

Problem definition: Users do not see the filters on our search results page.

Data validation: Less than 15% of users use filters when searching for products, which is very low compared to industry standards.

Solution

Next, propose a solution. Describe the change so someone who reads the hypothesis can understand the change, without screenshots. Then, add a rationale that offers a theory about why this solution is the right one to solve the identified problem.

Proposed solution: Move the filters to the left side of the results.

Rationale: This is the most commonplace for filters to be; users are more likely to notice them there.

Result

Finally, predict a result that ties your hypothesis back to your key business metrics. Include metrics that determine the success or failure of your experiment. Once you know those, decide on the specific metrics you will use to track success in the experiment or campaign.

Primary metric: % of users who use filters will increase

What does success look like: % of users who move on to a product page will increase; purchase rate will increase.

Examples of strong hypotheses

Example 1

 

Problem

Users find the featured products on the homepage irrelevant. Only 12% of users click on them. 9 out of 11 users state that they never found interesting products on the homepage.

Solution

Set the algorithm for featured products to display products from recent categories the user has visited. If the user visits the category, they have expressed interest in those types of products.

Result

Percentage of users that click on the featured products and percentage of users that added a product to cart will increase.

What makes this strong?

  • Qualitative and quantitative validations to the problem

  • Solution is based on a common UI practice in the industry 

 

Example 2

 

Problem

Users do not understand the name on the tab of our financial services (“money services”), as explained in usability interviews. Therefore, 85% of users drop off after getting to the page.

Solution

Try different names for the sections such as: “financial services,” “professional services,” or “money."

Result

More clicks on the tab and smaller drop-off rate on the section landing page.

What makes this strong?

  • Clear problem identification, including symptom 

  • The results encapsulate quantity (more clicks to section) and quality (less drop off at section)

Best practices

Engage in discovery

One key difference between a well-formulated hypothesis and a guess is data. Build a business intelligence report. Dive into existing data sources and carefully observe your customer’s journey through your site with direct data sources like web analytics data and indirect data sources like competitor overviews.

Use data that is strongly linked to your company’s goals to ensure you are focusing on areas of significant impact rather than making UX changes in isolation. How does your company define success on the web? Data sources can help you validate the problem and the solution rationale. Avoid generating hypotheses simply based on intuition.

Create a testable hypothesis

To efficiently test your hypothesis, identify the metrics to track and define a clear criteria for success and failure. 

For example, you hypothesize that removing breadcrumb navigation from the checkout page will help visitors stay in the funnel and increase conversions. The difference between the original and the variation is the presence or absence of breadcrumbs. The effect of that change can be measured in the number of conversions.

Use insights as a learning opportunity

Hypothesis-driven experimentation will give you insight into your visitors' behaviors. These insights generate additional questions about your visitors and site experience, and drive an iterative learning process.

Ideally, the cycle follows this general pattern:

  1. Gather data about your visitors' behaviors and industry, and use insights from that data to ask questions.

  2. Formulate a hypothesis based on insights from your data.

  3. Design and implement an Experiment or Campaign based on your hypothesis.

  4. Analyze your results to decide whether your hypothesis is confirmed or rejected.

  5. Create and document conclusions.

  6. Use conclusions to create new questions.

Connect to your company’s problems

Begin your process with a problem, not a solution. If a solution fails to deliver the result you expect, use the problem statement to explore new potential solutions and iterate. This practice enables you to focus on the problems that prevent your company from reaching its goals.

Learn about connecting your optimization program to company goals.