Example experiment design for travel sites

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Performance Edge
  • Optimizely Feature Experimentation
  • Optimizely Full Stack (Legacy)

One of the challenges in experimentation is designing a good test. A well-designed experiment can promote your site experience and improve conversion rates; a poorly designed one can stall your program. Once you have a hypothesis, how do you turn it into an experiment generates real insight and ROI?

This article walks through an impactful experiment that Hotwire, a leading travel bookings company, ran on its mobile website. Use this scenario to learn about designing experiments and explore ideas for your own site.

Although the example experiment uses Web Experimentation, you can implement a similar experiment using Optimizely Feature Experimentation or Optimizely Full Stack (Legacy).

Hotwire's Car Rentals homepage

See more customer stories and experiment ideas for travel sites.

Story

Hotwire is a travel company that provides flight, hotel, and car bookings through its website and mobile app. Since travelers' research and reservation needs are uniquely mobile, their mobile app and mobile website are important parts of the business.

Hotwire’s Site Optimization team identified the mobile web experience as a prime candidate for redesign. The experience had not been updated in several years, and the design team saw an opportunity to make the car rental product pages more seamless for customers on-the-go.


Original
Mobile-optimized website variation

They designed to test a major redesign that took a big step away from the original experience and baseline.

Setup

Hypothesis

"If we refresh the design of our car rentals mobile web product, then we can increase the conversion rate because we can provide a better visual and interaction experience to our mobile users." The Hotwire designed the variation to be more consistent with their mobile app experience, which had already solved some of the pain points of browsing and booking from a mobile device.

Below we describe key elements they included to create an effective test.

Pages

Pages that this experiment included: 

  • Car rentals product detail pages.

  • Pages in the checkout funnel

    If you are using Optimizely Web Experimentation to test on a checkout page, you might need to configure your site for PCI compliance

  • Purchase confirmation page

Hotwire’s goal was to make it easy for customers to use the mobile web experience to rent a car. So, they created a variation that dramatically changed the Car Rentals product detail page and tracked outcomes through the funnel.

By thinking about the visitors' intent and experience on the website for mobile users, the team was able to focus on addressing pain points in this funnel. Their test is designed to answer a specific question: does a mobile web experience that is consistent with the mobile app help more customers move further down the funnel?

Audiences

The team targeted the experiment to mobile visitors. Visitors who viewed the website on a mobile device were eligible to see a variation of the Car Rentals product detail page.

The Hotwire team hypothesized that mobile visitors would be more likely to convert if the site experience is optimized for on-the-go browsing and booking behaviors. Layouts that allow for quick scans of critical details on small screens, an image of the car on the same page, and an option to continue at top of a long scroll might help mobile visitors evaluate their options more easily.

Events

Metrics help you measure differences in visitor behavior based on changes you make to your site in experiment variations. The event you choose as your primary metric determines which variations are winning and losing.

  • Primary metric – The improvement in click-throughs as visitors travel down the funnel helped Hotwire confirm that their new design is helping customers carry out their intent: to rent a car.
    • Clicks to the Continue button.
  • Secondary metrics and monitoring metrics – The Hotwire team prioritized final purchase metrics to track the downstream impact of the experiment. Clicks to the navigation elements at the top of the screen help the team confirm visitor intent and monitor the health of the funnel in general.

    • Clicks for final purchase.

    • Views of the purchase confirmation page.

    • Clicks other progress buttons, throughout the funnel.

    • Clicks to navigation elements like Back and Home.

Learn more about primary and secondary metrics. If you are experimenting in a purchase funnel, read about choosing between macro and micro-conversions in an experiment.

Share and evolve

The Hotwire team presented findings from their experiment to the company and started brainstorming ways to apply what they learned to Hotwire’s other lines of business.

Share your results with stakeholders is important. It helps other teams incorporate what you have learned into their work and raises the visibility of your program. When programs do not share their findings, they risk siloing important insights that may be key to the business’ success.

By sharing their findings, the Hotwire team helped others learn more about customers and target them effectively. This also allowed them to bring the fresh insights they generated to a new cycle of experimentation, and to other parts of the business.