Example experiment design for retail sites

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Performance Edge
  • Optimizely Feature Experimentation
  • Optimizely Full Stack (Legacy)

One of the challenges in experimentation is designing a good test. A well-designed experiment can promote your site experience and improve conversion rates; a poorly designed one can stall your program. Once you have a hypothesis, how do you turn it into an experiment generates real insight and ROI?

This article walks through an impactful experiment that Jawbone, a digital-first retailer of fitness trackers, ran on its homepage. Use this scenario to learn about designing experiments and explore ideas for your own site.

Although the example experiment uses Web Experimentation, you can implement a similar experiment using Optimizely Feature Experimentation or Optimizely Full Stack (Legacy).

jawbone's homepage

See more customer stories and tips for experiment design in ecommerce.

Story

Jawbone is a world leader in consumer technology and wearable devices, like fitness trackers. Its website does the heavy lifting of communicating the company’s vision and serving as the main ecommerce portal for consumers. 

When Jawbone’s optimization team learned that one of its fitness trackers had been chosen for Oprah Winfrey’s “Oprah’s Favorite Things” wish list, a huge celebrity endorsement, it saw an opportunity. Chris Schroeder, Product Manager for Testing and Conversion Optimization, decided to test and learn from a new audience: visitors who may never have heard of Jawbone or considered purchasing a fitness tracker before.

Setup

Hypothesis

Chris' team hypothesized that new visitors would be ready for an educational mindset, rather than ready to compare the features of one tracker against others. Highlighting the benefits of fitness trackers in general would reduce friction and make the decision to purchase easier for these visitors.

They proposed to run an experiment on the homepage that targeted new visitors on the day “Oprah’s Favorite Things” wish list was announced. The experiment would switch a product lineup on the homepage for a series highlighting overall benefits of the featured fitness tracker. If new visitors saw the benefit of purchasing a tracker, they would be more likely to enter the purchase funnel.

Original and variation of Jawbone's A/B test.

Below are key elements that Jawbone used to create an effective test.

Pages

Pages that this experiment included: 

  • The homepage

  • Product detail pages

  • The checkout funnel

  • Purchase confirmation page

Chris’ team was trying to educate new visitors to Jawbone’s site who may not have known much about fitness trackers, he created his variation on the homepage. He tracked events there and across the checkout funnel.

By thinking about the visitors' intent and experience from the homepage to the purchase funnel, Chris' team was able to focus on the pages that matter in this experiment. Their test is carefully designed to answer a specific question.

Audiences

Chris' team targeted the experiment to new visitors on the day when "Oprah’s Favorite Things" was published. Those visiting the site for the first time were eligible to see a variation of the homepage that showed the benefits of fitness trackers instead of a product lineup.

Chris' team hypothesized that new visitors would be in a more educational frame of mind, rather than ready to compare the features of one tracker with the others. Showing the benefits of a tracker would help those visitors evaluate just the one tracker that they were there to see. This would reduce friction and make the decision to purchase easier for those visitors.

Events

Metrics help you measure differences in visitor behavior based on changes you make to your site in experiment variations. The event you choose as your primary metric determines which variations are winning and losing.

  • Primary metric – The increase in clicks to the "Shop Now" CTA in the hero image confirmed that visitors wanted to learn about the benefits of fitness trackers and evaluate one product, rather than choose between several.
    • Hero image CTA clicks on the homepage: 126% increase
  • Secondary metrics and monitoring metrics – Chris’ team prioritized revenue per visitor as a secondary metric to track the downstream impact of the experiment—and ensure that they are moving the needle in the right direction. Clicks to the top navigation helped the team confirm visitor intent and monitor other parts of the funnel.
    • Revenue per visitor: 43% increase
    • Clicks to the top navigation on Fitness Trackers: 96% increase

Learn more about primary and secondary metrics.

Share and evolve

Chris presented findings from the experiment to the UI/UX team and highlighted opportunities to improve the performance of Jawbone’s product pages.

Share your results with stakeholders is important. It helps other teams incorporate what you have learned into their work and raises the visibility of your program. When programs do not share their findings, they risk siloing important insights that may be key to the business' success.

By sharing their findings, the Hotwire team helped others learn more about customers and target them effectively. This also allowed them to bring the fresh insights they generated to a new cycle of experimentation, and to other parts of the business.