Ecommerce: Tips for good experiment design

  • Updated
This topic describes how to:
  • Collect accurate data in your experiments
  • Avoid common mistakes in ecommerce experiment design

In ecommerce, it is important to pay attention to business cycles and visitor behaviors. Tests that do not consider these patterns risk losing important pieces of data from the Results page. Good experiment design can ensure that you capture an accurate reading of conversion rates and other visitor behaviors.

Do not make data-driven decisions without the full picture. Read on for tips on designing robust experiments that capture a full cycle of data for ecommerce.

Tip 1: Test through a full business cycle

Traffic and usage of a website often follows external patterns: holiday schedules, weather, and more. Often, the clearest traffic pattern for an ecommerce business is based on the days of the week.

Let us take the traffic to Optimizely's Knowledge Base as an example:

Notice the regular cycles of traffic. The Knowledge Base is typically used by professionals while at work, so site traffic consistently dips lower on weekends.

In ecommerce, a weekly usage pattern is the most common. Visitors often have more time to browse products and make purchase decisions on the weekend. Or, depending on the type of product, visitors might more typically browse on the weekend and purchase during a work break on a weekday. The weekly usage pattern can vary, depending on the business. 

Plan your experiment cycles according to your website's usage patterns. It is a good idea to run experiments for at least a full week to preserve the ratio of data you collect on weekends and weekdays (through the whole week).

Imagine you start an experiment on Friday and see statistical significance by Monday. You may be tempted to stop the test. Here is why you should not: your experiment spans two (2) weekdays and two (2) weekend days; but it leaves out three (3) weekdays. Weekend behaviors are disproportionately amplified in your data because your collection period does not cover the full cycle. Let the test run for three more days.

A weekly cycle is common for ecommerce businesses but your business might have bi-weekly, monthly, or other time-based cycles. Running a test for a full business cycle gives you a more accurate picture of the impact of your changes.

Tip 2: Consider time-to-purchase before stopping an experiment

Time-to-purchase measures the average time that lapses between a visitor's first site visit and a purchase event. In this period, visitors take time to get oriented, compare products and prices, and make up their minds. Before stopping an experiment, consider your average time-to-purchase. 

Set your traffic allocation to 0% so no new visitors can enter the experiment, but keep running it for the length of your average time-to-purchase.

Here is why. Imagine you are tracking purchases in an experiment. Once you get statistical significance, you stop your test immediately. Visitors who enter the experiment near the end do not have time to complete their purchase before you stop tracking. You lose data for visitors who end up buying, but whose purchases are not captured on your Results page. 

Closing the experiment to new visitors but capturing final conversions will help you get more accurate data on your conversion rate and the impact of your changes.

Time-to-purchase varies for different products and audiences, so you need data that is specific to your website and products. Many analytics platforms can track average time-to-purchase. Here is a guide to tracking time-to-purchase in Google Analytics.

In general, incorporate information about your site's business cycles and visitor patterns into your experiment design. This will help you design robust experiments and build a rich, accurate, data-driven picture of your visitors' responses to the changes you test on your site.