- Design good experiments using five key elements of effective tests
- Explore ideas for your own site
One of the challenges in experimentation is designing a good test. A well-designed experiment can promote your site experience and improve conversion rates; a poorly designed one can stall your program. Once you have a hypothesis, how do you turn it into an experiment generates real insight and ROI?
At Optimizely, we work with the best experimentation programs in the world. We have learned a lot about experiment design and have identified five key elements that most successful experiments include.
This article walks through an impactful experiment that Jawbone, a digital-first retailer of fitness trackers, ran on its homepage—and breaks down five components that were key to its success. Use this scenario to learn about designing experiments and explore ideas for your own site.
Story
Jawbone is a world leader in consumer technology and wearable devices, like fitness trackers. Its website does the heavy lifting of communicating the company’s vision and serving as the main ecommerce portal for consumers.
When Jawbone’s optimization team learned that one of its fitness trackers had been chosen for Oprah Winfrey’s “Oprah’s Favorite Things” wish list, a huge celebrity endorsement, it saw an opportunity. Chris Schroeder, Product Manager for Testing and Conversion Optimization, decided to test and learn from a new audience: visitors who may never have heard of Jawbone or considered purchasing a fitness tracker before.
Setup
Hypothesis: Chris’ team hypothesized that new visitors would be ready for an educational mindset, rather than ready to compare the features of one tracker against others. Highlighting the benefits of fitness trackers in general would reduce friction and make the decision to purchase easier for these visitors.
They proposed to run an experiment on the homepage that targeted new visitors on the day “Oprah’s Favorite Things” wish list was announced. The experiment would switch a product lineup on the homepage for a series highlighting overall benefits of the featured fitness tracker. If new visitors saw the benefit of purchasing a tracker, they would be more likely to enter the purchase funnel.
Below are key elements that Jawbone used to create an effective test.
|
Every test you run in Optimizely Experimentation is built on top of pages. When adding pages, think carefully about the paths that your visitors take. What is a visitor in a given funnel trying to accomplish? What problem are you trying to solve for them? Pages that this experiment included:
How did this help?
A considered approach to pages helps you focus on the visitor’s journey when designing your experiment. Because Chris’ team was trying to educate new visitors to Jawbone’s site who may not have known much about fitness trackers, he created his variation on the homepage. He tracked events there and across the checkout funnel. By thinking about the visitors’ intent and experience from the homepage to the purchase funnel, Chris’ team was able to focus on the pages that matter in this experiment. Their test is carefully designed to answer a specific question. |
|
Chris’ team targeted the experiment to new visitors on the day when “Oprah’s Favorite Things” was published. Those visiting the site for the first time were eligible to see a variation of the homepage that showed the benefits of fitness trackers instead of a product lineup. How did this help?
In Optimizely Experimentation, audiences are optional, but you are far more likely to reach statistical significance with an audience, so we recommend that you add one. Adding audiences forces you to focus on your target market and design experiences for them. Who are you improving this experience for? What are the priorities of that visitor segment? What do you want to learn about those visitors’ expectations and behaviors? Chris’ team hypothesized that new visitors would be in a more educational frame of mind, rather than ready to compare the features of one tracker against the others. Showing the benefits of a tracker would help those visitors evaluate just the one tracker that they were there to see. This would reduce friction and make the decision to purchase easier for those visitors. |
|
Integrations help you leverage a variety of different digital tools to get a more in-depth understanding of your users. It is important to link your systems so you can track user behavior. How did this help?
The most successful experimentation programs leverage Optimizely’s integrations. By integrating Optimizely Experimentation with other technology platforms that your company uses, you not only maximize the information that you are getting from different data sources—you gain the ability to segment results for more meaningful business insights. Not setting up integrations holds your program back from deeper learning and the ability to connect other parts of your analytics landscape to your experimentation efforts. |
|
Metrics determine whether your experiment “wins” or “loses.” It is easy to create and add metrics in Optimizely Experimentation, but the strategy behind an experiment’s metrics is key its success. Primary metric:
Secondary metrics:
How did this help?
Primary metric: The most important decision you make when designing your experiment is choosing the primary metric. It is the metric that determines the speed of your test—and whether it wins or loses. It can be tempting to choose revenue, but revenue is rarely an effective primary metric because it is generally influenced by many factors. The best primary metric is the action that you want to directly influence with your experiment. Always solve for the primary goal first. The increase in clicks to the “Shop Now” CTA in the hero image confirmed that visitors wanted to learn about the benefits of fitness trackers and evaluate one product, rather than choose between several. When choosing secondary metrics, you often balance site traffic and complexity. Optimizely Experimentation calculates statistical significance for the primary metric separately to ensure that it always reach statistical significance at top speed, so you can make important business decisions. But for all other goals, the more you add, the more visitors you need to determine a win or loss. If traffic is a limited resource for your program (it is for most), you must balance it against goal complexity. Chris’ team prioritized revenue per visitor as a secondary metric to track the downstream impact of the experiment—and ensure that they are moving the needle in the right direction. Clicks to the top navigation helped the team confirm visitor intent and monitor other parts of the funnel. Learn more about primary and secondary metrics. |
|
Chris presented findings from the experiment to the UI/UX team and highlighted opportunities to improve the performance of Jawbone’s product pages. How did this help?
Sharing the results of your experiments is important. It helps other teams incorporate what you have learned into their work and raises the visibility of your program. When programs do not share their findings, they risk isolating important insights that may be key to the business’ success. The team has helped Jawbone transform into an optimization-led company with a strong experimentation culture. The testing team has begun to influence the design process in earlier stages and continues to experiment with the website’s retail experience. Here are a few templates for sharing your findings. |
<%= heading %>