The Experiment Results page

Summary

The summary provides a high-level overview of the experiment. It allows you to compare how each variation is performing for the primary metric compared to the original.

Here is what you see once visitors enter your experiment:

Optimizely Web Experimentation and Optimizely Feature Experimentation shows the number of unique visitors bucketed into each variation.

In the previous screenshot, 423 visitors (or 32.59% of visitors in this experiment) have seen the original variation, 410 (31.59%) have seen Variation #1, and 465 (35.82%) have seen Variation #2.

Metrics

Below the summary, you will see the results for each metric that you added to your experiment. The primary metric is always at the top and expanded by default. Secondary metric results will be automatically collapsed. Select the arrow next to the secondary metric name to expand its results or select Expand all to view all secondary metric results. 

• Unique Conversions or Total Conversions – When you add a metric to an experiment, you choose unique or total conversions. Unique conversions show deduplicated conversions, so a single visitor who triggers the same event multiple times is counted just once. Total conversions show a simple total of conversions for the event.

• Conversion Rate or Conversions per Visitor – Under the Unique conversions view, Optimizely Web Experimentation and Optimizely Feature Experimentation show the conversion rate: the percentage of unique visitors in the variation who triggered the event. Under the Total conversions view, you will see Conversions per Visitor: the average conversions per visitor for visitors in the variation.

• Improvement –

  • Optimizely Web Experimentation and Optimizely Feature Experimentation display the relative improvement in conversion rate for the variation over the baseline as a percentage for most experiments. For example, if the baseline conversion rate is 5% and the variation conversion rate is 10%, the improvement for that variation is 100%.

  • Optimizely Web Experimentation and Optimizely Feature Experimentation display the absolute improvement for any experiment where Stats Accelerator is enabled and for Multi-armed bandit (MAB) optimizations.

• Confidence interval – The confidence interval measures uncertainty around improvement. Stats Engine provides a range of values where the conversion rate for a particular experience lies. It starts wide, and as Stats Engine collects more data, the interval narrows to show that certainty is increasing.
  
  Once a variation reaches statistical significance, the confidence interval always lies entirely above or below 0.

Statistically significant and positive	Statistically significant and negative	Not yet conclusive

• Statistical significance – Optimizely Web Experimentation and Optimizely Feature Experimentation show you the statistical likelihood that the improvement is due to changes you made on the page, not chance. Until Stats Engine has enough data to declare statistical significance, the Results page will state that more visitors are needed and show you an estimated wait time based on the current conversion rate.

  Multi-armed bandit optimizations do not calculate statistical significance.

Graphs

You can toggle between different graphs for each metric. To see or hide charts, click Hide graph or View graph.

• Improvement over Time (the default) – Improvement in this metric for each variation, compared to the baseline. Improvement is relative improvement. Not cumulative. Refer to Confidence intervals and improvement intervals for more information.

• Visitors over Time – Total visitors for the variations and the baseline

• Conversions over Time – Conversions per day in this metric for each variation, including the original

• Conversion Rate over Time – The cumulative conversion rate for each variation, including the original

• Statistical Significance over Time – Cumulative statistical significance for the variation

Filter by date range

Use the Date Range dropdown to select start and end dates for your Results page view. Then, click Apply. The results generated will be in your computer's time zone.

Segment experiment results

By default, Optimizely Web Experimentation and Optimizely Feature Experimentation show results for all visitors who enter your experiment. However, not all visitors behave like your average visitors. Segmenting your results is a powerful way to gain deeper insights into your customers to design data-driven experiments and personalization campaigns. 

Use the Segment dropdown to drill down into a segment of your visitors. You can segment by a single attribute value or a combination of attribute values (learn more about using combinations).

For Optimizely Web Experimentation, the default segments are:

• Browser – Firefox, Google Chrome, Internet Explorer, Opera, Safari, Unknown

• Source – Campaign, Direct, Referral, Search

• Campaign

• Referrer

• Device

You can also segment by up to 100 custom attributes in Optimizely Web Experimentation or Optimizely Feature Experimentation.

Change the baseline

Sometimes, you may want to see how all your variations compare to one variation in particular—which may not be the original. Use the Baseline dropdown to select a different variation as the baseline.