Take action based on the results of an experiment

Multiple winning variations

You may see multiple winning variations in a single test. This is a good outcome, but what do you do with all of these winning variations?

• Combine the ideas – Try combining multiple winning variations into a single optimized experience. By combining your winning ideas, you optimize and iterate based on multiple data points at once. When you are trying to test what resonates with visitors the most, mix existing themes in new combinations to find the best experience.
• Expand on one idea – If you see a promising trend in one of your variations, consider taking a little time exploring that opportunity. 
• Implement the highest winning variation – When speed is a concern and you want to implement results quickly, you can choose to implement the highest-winning variation in your code. But before you do, segment your results to check how your most valuable customers respond to the change. If those important visitors prefer the next-winning variation, weigh the business impact of those two choices. 

Losing variation

Some teams try to avoid losing tests, but experiments with losing variations are not bad. They are often just as actionable as winning variations, and they provide focused and valuable insights about your visitors' behaviors that help to guide your optimization efforts in the long run. The most valuable result of A/B testing is learning more about the customer.

Pay attention to your losing variations. Dig deep into the results of losing tests to find out:

• Why do visitors who see this experience convert less often?

• Do certain visitors convert significantly better (or worse) than others?

• Can you use this data to brainstorm new hypotheses about how visitors respond to your site experience?

  • What can you optimize, based on this information?

  • How can you leverage this insight? 

Use what you have learned to plan a new, bolder experiment or make major decisions about the direction of a redesign.

Here are a few more ways to handle a losing variation, after you analyze it:

• Try it again, with some attitude – If you have generated a statistically significant result, even if that result is negative, you have still shown an ability to affect your visitors' behavior with your experiment.
• Move on – Sometimes, the thing you are trying to test is already relatively "optimized" (though no site is ever completely optimized). Continuing to test the same thing will not only generate more negative results, but you will also encounter an additional opportunity cost for not testing something with clearer potential to generate positive results. This might be true if you have already iterated multiple times on the same experiment idea, or if you have been running tests on one area of your site for an extended period.
• Sometimes a loser, is actually a winner – Sometimes, you will have the case where a losing variation actually drives advantageous business results. For example, if you test a prototype of a costly redesign, and it loses, you can avoid committing to the full project. Prototyping and testing can generate benefits by avoiding costly investments.

Inconclusive experiment

Sometimes, when you check your experiment at the end of the projected time-to-results, your test has not reached statistical significance. You know that inconclusive tests can provide valuable information. Waiting longer will help you gather more data—but how long should you wait? What is the potential impact of this test, compared to the next experiment you might run?

Inconclusive tests are the most confusing type of test result. But they also generally point in a clear direction: go bigger!

Many test executions simply are not dramatic enough to impact results. Organizational fear and a lack of resources are the biggest hurdles to testing more significant changes.

Here are a couple of ways to go bigger:

• Test more than one element at a time – Although you may sacrifice some empirical rigor by testing more things at once, your single greatest responsibility is to move the needle; you can focus on getting better at establishing why things are happening once you have gotten that lift you have been looking for.
• Increase the degree of drama – Make your variations clearly and visibly different. Making more substantial attempts at changing the environment will yield proportionally stronger results. 

In certain situations, you may wish to take action on an experiment that has run for the projected time but has yet to reach significance. Taking action is a business decision based on the Results page data and your business goals. Below, we discuss a few tactics that may help.

Check Visitors Remaining

Visitors remaining estimates how long your experiment needs to run to reach significance, based on traffic, conversion rate, and the number of variations. Specifically, visitors remaining estimates the number of visitors needed for a particular variation. If your conversion rates for the original and that variation stay the same, you would need X number of visitors for Stats Engine to call a significant result.

But, if the conversion rates start to change, your visitors remaining will also adjust. A ballpark estimate based on visitors remaining will help you make a business decision on how long to wait.

To decide whether to keep running the test, compare the visitors remaining to the number of visitors projected in your experiment test plan.

Wait for statistical significance if:

• You have not reached the number of visitors predicted in your experiment test plan.

• Your visitors remaining suggests you do not have long to wait.

Declare the test inconclusive if: 

• You have exceeded the number of visitors you planned for, and visitors remaining suggests you will not reach significance anytime soon.

Imagine, for example, that you want to be confident that the 5% lift you see is real. And you would likely have to wait another two weeks to gather enough data to say that the likelihood that lift is false is less than 5% (or 95% statistical significance). You could keep running the test—but it is probably not worth it. Together, the data for visitors remaining, your experiment test plan, and the lift you expect to see already indicate that the change you made did not change your visitors' behavior in a significant way.

At this point, segment your results and check your secondary and monitoring metrics to look for insights for the next round of tests.

When visitors remaining does not provide a ready answer, check the confidence interval to help you interpret your results.

Check the Confidence Interval

The Confidence Interval can help you decide whether to keep running a test or move on to another idea. The interval always straddles zero for experiments that have not reached significance yet.

As Stats Engine gathers more data from visitors' behaviors, the confidence band expands and contracts. The more erratic visitor behaviors are, the less sure Stats Engine is about future results, and the interval widens. A narrow difference interval indicates that Stats Engine is honing in on the likelihood that the variation will convert at the same rate in the future. 

If your results are inconclusive, your difference interval can provide insight when paired with visitors remaining.

Segment Your Results

Segments are one of the analyst's most important tools for digging into overall performance metrics. Your total performance can be thought of as an average population, but different sub-groups may have different goals—and different conversion rates. Dig into your segments to discover whether there is a group that responds to your experiment differently from the average population. If there is a segment that responds differently to your experiment, that may lead you to develop a personalization strategy.