10 common experiments and how to build them

When you start experimenting on your site, you may not know what to test first. You can use these common experiment examples to learn about setting up experiments and explore ideas for your own site.

See Six steps to create an experiment in Optimizely Web Experimentation to build experiments like the ones below.

What not to test

Experimentation is both an art and a science. Here are three common mistakes to avoid if you want your tests to make an impact:

1. Testing parts of your site that get low traffic – If you have a low-traffic site, see Testing tips for low traffic sites. Avoid running experiments on low-traffic pages. It takes longer to get as many visitors into the experiment as you can get on a high-traffic page, and you may not see statistical significance while the test stalls. This includes remote pages of your site as well as parts of a page most visitors do not see, like the footer of a high-traffic page or a drop-down menu.
2. Testing changes that are too small to matter – You may want to test small changes at first because you are not yet sure what impact your testing will have or how visitors will respond. This makes a bold change more valuable because you learn more about your visitors' behavior. A bigger change also means your test can run faster since it makes more of an impact. If you test a change that is too small, you may not see significance, and your program stalls while you wait.
3. Testing ideas that are too far removed from core goals – For most programs, what you test on your website relates to three major goals: increasing engagement, collecting data about your visitors (such as name, email, preferences), and purchases. Experiments that do not drive towards the core business goals for your company do not help your program make an impact. Focus your testing on changes that affect metrics that matter.

Idea 1: Optimize for geographical differences

Neha's market research shows the top concerns of her North American and European customers differ from each other.

• In the U.S., her product's biggest selling point is the flexible, developer-friendly experience it offers.
• In European markets, most sales conversations start out with questions about privacy and security.

Neha wants to experiment with different messaging on her site for European customers versus U.S. customers. She thinks that security-focused branding can make her product more compelling for European consumers.

Hypothesis

The product's value proposition focuses on the developer experience, which is a selling point for U.S. customers. This misses the European companies' top concern - security. If European visitors see a security-focused value proposition in the hero banner of a high-traffic landing page, the number of leads from that region will increase.

Pages

The URL for the landing page

Audiences 

Visitors in Europe

Metrics

• Clicks to View Pricing (primary)
• Clicks to Explore the Docs (secondary)
• Pageviews for the pricing and form pages (secondary)
• Clicks to the form submission button (secondary)

Editor

Original: 

Variation: 

1. Select the variation.

2. Click to select the value proposition text.

3. Enter a new value proposition under HTML. Click Save.
4. QA with the Preview tool.
5. When you are ready, click Start Experiment.

Technical skills

None

Interpret results

• Win – Neha's hypothesis is correct. Security-focused value messaging increases conversions from visitors in Europe. She can then expand this branding across the site. 
• Lose – The results do not support Neha's hypothesis, and she must determine why. Do visitors in Europe who land on this page still value developer tools over security? Maybe developers, not program managers, make decisions based on this page. She can analyze her data and apply her findings in future tests.
• Inconclusive – Tells Neha that her hypothesis is incorrect, or the test is not distinct enough. Visitors continued to behave in the same way. She could try a more radical change, like switching out all the copy and imagery on the page.

Idea 2: CTAs for new visitors versus subscribers

David manages the landing page for a streaming video site. It features a call to action (CTA): "Join free for a month." All visitors see this page, but research shows that subscribed customers find the button confusing because they have already joined. Because David's company cares about logins, he wants to test a separate CTA for subscribers.

Hypothesis

Changing the Join Free CTA button to match the subscribed visitor's mindset can increase the number of logins by reducing their confusion. 

Pages

The URL of the landing page.

Audiences

Returning visitors

Metrics

• Clicks to the CTA button (primary)

Editor

Original: 

Variation:

1. Select the variation.
2. Click the text on the CTA to select it.
3. Edit or replace the text with your new CTA. To remove extra text, click to select. Under Visibility, click Remove.
4. Click Save.
5. QA with the Preview tool.
6. Click Start Experiment when ready.

Technical skills

None

Interpret results

• Win – The results support David's hypothesis: Returning visitors respond well to a CTA about seeing what is next. Next, he can expand on this theme or test variations that interest visitors. If he has Personalization, he can feature a show that the visitor has watched or highlight popular shows for an audience segment.
• Lose –The results do not support David's hypothesis. He can see if he has made any incorrect assumptions. Are returning visitors looking to do something different from what he thought? Is there something else on the page that influences their decision to log in? A loss can give him insight into how subscribed customers react to this page.
• Inconclusive – suggests that the hypothesis is off, or the change is not enough—visitors continue to behave in the same way. He can try clearer, more distinct messaging or move on to a more urgent experiment.

Idea 3: Remove distractions from the checkout funnel

Jessica’s team has a mandate to improve the purchase rate of her company’s retail site. They looked at the data and noticed that a high percentage of customers leave the checkout flow through the breadcrumb navigation. Jessica believes that she can improve the purchase rate by removing distractions from the checkout funnel. She wants to validate this idea by testing a less cluttered navigation.

Hypothesis

Navigation elements distract customers from completing their purchases. Removing the distractions can increase the purchase rate by 7%.

Pages

Every URL in the checkout flow. You can use substring match if all your checkout pages follow a pattern like yourdomain.com/checkout.

Audiences

Everyone

Metrics

• Clicks to Continue on each step of the checkout (primary)
• Pageviews for the purchase confirmation page (secondary)
• Clicks to Submit Order (secondary)
• Pageviews for each step in the checkout flow (secondary)

Editor

Original:

Variation:

1. Select the variation.
2. Go to Changes > Create.
3. Click Element Change and select the elements you want to remove, such as the container that includes the breadcrumb navigation. Click the Hidden icon to hide the feature without removing the field or click the Remove icon. Click Save.
4. QA with the Preview tool.
5. Click Start Experiment when ready.

See Cannot load a password-protected, gated, or session-specific page in the editor if you have trouble loading your page in the Editor.

Technical skills

None

Interpret results

• Win – Jessica’s hypothesis is correct. Visitors leave the funnel because they have the opportunity, not because they do not intend to complete their purchase. Jessica can then shorten the checkout flow or remove other distractions from the funnel.
• Lose –The team’s assumption about visitors dropping off is not correct; visitors may be leaving on purpose. They may see if there is a negative or confusing experience during checkout.
• Inconclusive – Suggests that the hypothesis is off, or the change is not enough—visitors keep leaving at approximately the same rate. Jessica may investigate where, how, and why visitors leave the funnel.

More resources

• Split Test Ideas for Urban Outfitters’ Checkout Funnel

Idea 4: Rearrange subscription price-points

Emmanuel has noticed that the mid-priced Plus plan on his page receives a surprisingly high percentage of subscribers, compared to the free Basic and upgraded Premium plans. This distribution also matches the click-through rate to the plans details pages. Emmanuel thinks that the position of plan options affects interest and purchases. He wants to test a different arrangement to encourage more customers to evaluate and subscribe to the Premium plan.

Hypothesis

A smaller-than-expected percentage of visitors click the Basic or Premium plans on the pricing page due to positioning. Moving the Premium plans to the center of the page can increase interest and subscriptions.

Pages

The URL for the plans page

Audiences

Everyone

Metrics

• Clicks to Go Premium (primary)
• Clicks to Sign up and Get Plus (secondary)
• Plan details page pageviews (secondary)
• Purchase confirmation pageviews (secondary)
• Revenue (secondary)

Editor

Original: 

Variation: 

1. Select the variation.
2. Go to Change > Create.
3. Click Element Change and select the container for the highlighted element.
4. Click the Magnify icon under Rearrange. Then, click the element you want to rearrange the first element in relation to (the Plus container, in the example above).
5. Use the drop-down list below Rearrange to re-position the element. Click Save.
6. QA with the Preview tool.
7. Click Start Experiment when ready.

Technical skills

None

Interpret results

• Win – If more visitors click Go Premium in the variation, Emmanuel's assumption is correct that visitors are sensitive to the position of the packages on his pricing page. His results also support the idea that visitors are more willing to consider the Premium package. He can keep experimenting with ways to make the Premium package more visually prominent.
• Lose – The results do not support Emmanuel's hypothesis. He may not have identified the right problem or solution. If his goal is to increase Premium subscriptions, he can investigate what customers like about this package and test ways to draw attention to those value propositions. 
• Inconclusive – The change is not big enough or customers are not very sensitive to the position of plans. Emmanuel can use this insight in future designs.

Idea 5: Highlight key value propositions

Kate's company offers a number of pricing packages on the site, but research shows that customers find this list overwhelming and have trouble choosing. Based on this information, she wants to experiment with highlighting key value propositions to help customers choose between price points and packages.

Hypothesis

Customers are confused by the packages offered. Highlighting a key value proposition can make it easier for customers to choose, leading to an increase in purchases for that package.

Pages

The URL for the Plans page

Audiences

Everyone

Metrics

• Clicks to the highlighted package (primary)
• Clicks to Add-to-Cart (secondary)
• Pageviews for the package details pages (secondary)
• Pageviews for a purchase confirmation page (secondary)

Editor

Original:

Variation:

1. Select the variation.
2. Go to Changes > Create.
3. Select Insert HTML to add a value proposition next to a package. Insert your text in relation to another element on the page—in this case, the package. Click the Magnify icon, then select the package to highlight.
4. Enter your text under HTML. Under HTML Placement, reposition the inserted text as desired. 
5. Click Save. 
6. QA with the Preview tool.
7. Click Start Experiment when ready.

Technical skills

None

Interpret results

• Win – Kate confirms that highlighting value propositions helps customers choose and purchase a package. She can also check her secondary metric for overall purchases or overall Add-to-Cart clicks to ensure that this plan increases her company's revenue overall. If revenue decreases, she may want to see which plan has a larger revenue percentage and test compelling value propositions for that plan type. 
• Lose – The results do not support Kate's hypothesis. She has not identified either the right problem or the right solution. Maybe a different type of value proposition would work better: price per unit if her customers are especially price-conscious, or CTAs for buyer personas if she learns that certain personas have unique packages preferences.
• Inconclusive – The change may not be big enough. Kate can consider experimenting her way through a radical redesign of the pricing page.

More resources

Case study: Seeking the global maximum

Idea 6: Symmetric messaging

Wesley’s company is advertising a holiday sale on fitness trackers with two different ad campaigns: one focusing on holiday gift-giving and the other on outdoor adventures. His task is to optimize the paid search campaigns. Wesley believes he can quickly increase conversions by customizing the imagery to match each specific ad’s messaging, rather than using a generic page for all ads.

Hypothesis

The current landing page for Wesley's site promotes holiday gift-giving. When visitors click ads about a fitness tracker for outdoor adventures, they should see related imagery when they land on the site itself. If he switches the imagery to match the ad the visitor clicked, the site will align with the visitor’s mindset and goals, and conversions can increase.

Pages

The URL of the landing page

Audiences

Visitors who match the ad campaign audience condition.

The ad campaign audience condition is sticky, meaning a visitor sees the same experience every time they come back to the site. Visitors who see the adventure imagery will keep seeing mountains. Visitors who see gift-giving imagery will continue to see presents, even if they visit the page without the query parameter.

Metrics

• Clicks to Shop the Sale (primary)
• Clicks to product detail page below (secondary)
• Clicks to top-level navigation (secondary)

Editor

Original:

Variation:

1. Select the variation.
2. Click the image to select the container.
3. Click Background and upload your new image.
4. QA with the Preview tool.
5. Click Start Experiment when ready.

Technical skills

None

Interpret results

• Win – Wes confirms that symmetric messaging works well for his visitors and that the outdoor adventure messaging resonates. He can use this information to design a more differentiated landing page or create a Personalization campaign to deliver more targeted messaging.
• Lose – Wes learns that his outdoor adventure imagery does not resonate with visitors who clicked his ad. He may want to investigate whether they are looking for a different type of adventure and if the holiday messaging is still more powerful for these visitors.
• Inconclusive – Wes can try a more radical change if he thinks there is a good opportunity.

More resources

Case study: Testing personalized experiences at Secret Escapes

Blog: The impact of symmetry in online marketing

Idea 7: Personalize based on cookies

Silvia knows that her travel site adds cookies that she can leverage to create a personalized experience. She hypothesizes that she can increase browsing behaviors and purchases if she customizes the site’s homepage based on visitors’ interests. For example, Silvia features a hero banner that highlights a destination related to the geographical area that a visitor frequently browses.

Hypothesis

Silvia thinks that visitors who browse certain locations are more likely to buy vacation packages in those areas. If she features destinations in that area in the hero banner for visitors who browse more than two pages, she can increase browsing behaviors and conversions.

Pages

The URL of the homepage

Audiences

Visitors who meet the audience condition for a unique cookie that Silvia can personalize, such as a cookie that tracks the geographical area of vacation destinations that a visitor most recently browsed.

Metrics

• Clicks to the promoted hero banner (primary)
• Pageviews for the promoted destination (secondary)
• Clicks to Search (secondary)
• Pageviews for other destinations in the area (secondary)

Editor

Original:

Variation:

1. Select the variation.
2. Click the image to select the container.
3. Click Background and upload your new image.
4. QA with the Preview tool.
5. Click Start Experiment when ready.

Technical skills

None

Interpret results

• Win – Silvia confirms that more personalized content improves browsing and purchase behaviors for her site. She can continue testing this concept by promoting vacation types instead of geographical areas. She can also start to deliver more granular Personalization campaigns, such as promoting amenities and services often purchased or browsed in certain geographical areas.
• Lose – The results do not support Sylvia's hypothesis. She may investigate why. Maybe her visitors prefer to start with a wide array of options instead of a targeted experience. Maybe many visitors are not primarily shopping by the geographical area—the price of the vacation is more important to them.
• Inconclusive – The experiment may not be bold enough. Silvia can try a lightbox or overlay for a more radical change.

More resources

Inspiration for website personalization

Idea 8: Test promotion formats

Sarah’s company offers a promotional price on a one-year subscription on its birthday each year. The offer generates a significant jump in new subscriptions, and research shows that even customers who see the offer without purchasing respond positively.

The offer is promoted through an email mailing list and a note in the header. This year, Sarah plans to experiment with making the anniversary promotion more visible. She decides to test whether a site-wide banner will increase subscriptions.

Hypothesis

Sarah knows that her company’s anniversary sale increases the number of one-year subscriptions. If she promotes the promotion with a site-wide banner instead of the usual promotion in the header, more customers will see it and sign up for a discounted subscription package.

Pages

Every page on the site

You can use substring match if all your pages follow a pattern like yourdomain.com/.

Audiences

Everyone

Metrics

• Clicks to the banner link (primary)
• Pageviews for subscription details (secondary)
• Pageviews for the subscription form (secondary)
• Clicks to Submit for the form (secondary)

Editor

Original: 

Variation:

Create (or ask your developer to help create) an Optimizely Experimentation extension: a template that helps you add custom features like carousels and banners without continuous developer support.

1. Go to Implementation > Extensions.
2. Click Create New > Using JSON.
3. Copy the JSON for the butterbar extension.
4. Click Create Extension.
5. Click More and enable the extension.

Next, add the extension to your variation.

1. Go to your variation.
2. Go to Create. When you scroll down, your butterbar extension appears under Create Options.
3. Select the butterbar and edit it as desired. Remove the other promotion.
4. QA your experiment.
5. Click Start Experiment when ready.

Technical skills

None (but you may need to know JSON to create other, similar extensions).

Interpret results

• Win – Sarah knows that this type of element creates more conversions than the header promotion they typically use. She can decide to use this for other promotional offers. She can also experiment with different messaging or offers for logged-in vs. logged-out users (with cookie-based audience targeting).
• Lose – The butterbar variation is less successful at driving conversions. This is an opportunity for Sarah to find out why. She can look into her data to see if there are differences between segments, check whether the butterbar displays incorrectly for some visitors or browser types, or try a different design or message. 
• Inconclusive – Sarah can try a bolder version of the butterbar. She may conclude that for this type of promotion, visitors respond similarly either way. She can use this insight for designing future header or banner options.

More resources

• Create reusable templates for custom features using Extensions

Idea 9: Optimize a form

Greg’s company relies heavily on landing pages to generate leads for the sales team. The company uses the form inputs to validate high-quality leads and tailor marketing efforts. Greg has noticed that completion rates have fallen as more fields are added. 

Greg believes that he can optimize the form by balancing the quality and quantity of information. He wants to experiment with the form to see if they can ask for enough information without adding extra friction to the funnel.

Hypothesis

Prospective customers experience the fields in the lead generation forms as friction. Removing extra fields to streamline the form can increase form completions by 5-10%.

Pages

The URL for the page with the form

Audiences

Everyone

Metrics

• Clicks to form submission (primary)
• Clicks to the navigation elements on the page (secondary)
• Clicks to other buttons on the form page (secondary)

Editor

1. Select the variation.
2. Go to Changes > Create.
3. Go to Element Change and select the field. Click the Hidden icon to hide the feature without removing the field or click the Remove icon. Click Save.
4. Repeat Step 3 to hide or remove the label.
5. QA with the Preview tool.
6. Click Start Experiment when ready.

Technical skills

None

Interpret results

• Win – The results support Greg’s hypothesis that form fields are friction points for visitors. He may want to experiment further by removing more fields or using placeholder text instead of field labels for a cleaner look.
• Lose – The results do not support Greg’s hypothesis. This tells him he has not identified the right problem or solution. He can choose to do more research to understand what assumptions he has made that this data contradicts. Maybe customers in his industry expect to provide a large amount of information, or something else is negatively affecting the form experience.
• Inconclusive – Greg may have tested too small a change; he can experiment with a more radical change by removing all but necessary fields. Greg could declutter the visual design and add or remove explanatory text.

More resources

• A radical redesign for a lead generation form generates a big impact

Idea 10: Add social proof

Byron's team created a business intelligence report to help generate higher-quality test ideas. The report highlights the compelling impact of social proof. They do not currently feature any social proof on the site, but Byron knows the marketing team recently gathered a set of customer use cases and testimonials. He wants to experiment with adding social proof to a high-traffic spot to see if it helps customers convert.

Hypothesis

The payroll feature is relatively new and unproven in the market, but it is showcased and gets significant traffic. Adding social proof to the page shows visitors more evidence of success and can increase conversions on the Get Started button.

Pages

The URL of the feature page

Audiences

Everyone

Metrics

• Clicks to Get Started (primary)
• Clicks to the details page of the feature (secondary)
• Pageviews for the details page (secondary)
• Form submissions (secondary)

Editor

Original:

Variation:

1. Select the variation.
2. Go to Changes > Create.
3. Select Insert HTML to add a testimonial. 
4. Insert your text in relation to another element on the page. Click the Magnify icon, then click the selector to position the new text. 
5. Enter your text under HTML. Under HTML Placement, reposition the inserted text as you like.
6. Click Save. 
7. QA with the Preview tool.
8. Click Start Experiment when ready.

Technical skills

None

Interpret results

• Win – The results support Byron's hypothesis. He knows that social proof is compelling to his visitors and helps increase the conversion rate. He can build on this by adding a short video to boost the impact of the testimonial. If he knows the industry of some of his visitors, he can personalize the page with testimonials from businesses in the same one. He can also look for other opportunities on the site to experiment with social proof.
• Lose – The results do not validate his hypothesis. He can segment his results and check whether the variation performed better or worse for any particular groups of visitors.
• Inconclusive – The change Byron tested may be too small. He can run a low-effort test that makes a bigger impact. For example, he can add a picture to his testimonial to make it more visually compelling. He can find a more dramatic quote or feature the testimonial above the fold.

More resources

• comScore increases new leads by 69% with social proof