Experiment analysis

  • Updated

In Optimizely, experiments are structured tests designed to compare different variations of a webpage, feature, or experience to determine which performs best based on specific goals. These experiments help businesses optimize user experiences, increase conversions, and make data-driven decisions. All experiments created within Optimizely are visible on the Experiments page in Optimizely Warehouse-Native Experimentation Analytics.

Optimizely Analytics currently only supports A/B tests and multivariate tests as experiment types.

Optimizely offers different types of experiments for various objectives. Some experiments focus on learning and long-term insights, such as A/B tests, multivariate tests, and Stats Accelerator, which incorporate statistical significance calculations. Others prioritize immediate impact, like a multi-armed bandit (MAB) and the contextual bandit, which optimize for short-term gains without performing statistical significance analysis. Learn more about distribution modes and experiment types.

Access experiments

Go to Experiments to see a quick overview of the latest experiments. 

You can refine the list of experiments by adjusting the filters (such as date range) or selecting one or more Types.

Click any experiment to edit the associated scorecards. 

access-experiments.gif

Experiment results

Each experiment scorecard has a Summary and Explore tab. Learn how to access Experiments.

The Summary tab has key insights from the selected experiment to support decision-making.

The Explore tab lets you further analyze data within the experiment and its variations.

The following options are available on the Explore tab: 

explore-options-bottom.png

Summary

The Summary tab overviews the exploration, including the selected experiment, configured decision-making metrics, and the experiment results in the visualization window. Use graphs, date ranges, attributes, and the baseline to determine results. You can modify the scorecard configuration within the tab. Learn more about creating scorecards.

experiments-module-nav.png

Edit experiment

You can change experiment settings and see the experiment within Optimizely. Click the external link icon to go to the Flags section in Feature Experimentation.  

experiments-module-flags.png

Change the baseline

The Baseline option lets you compare your variations against a specific one instead of the original. To do so, select your preferred variation from the Baseline drop-down list.

Screenshot 2025-03-13 at 8.39.14 PM.png

Manage Metrics

You can add a new primary or guardrail metric and remove or edit previously added metrics.

adding-metrics.png

Visualization options

Within Summary > Visualization, you can select segmentation options (Segment and Group By) and graph options (Improvement Over Time, Results Over Time, and Statistical Significance Over Time). You can toggle between different graphs for each metric. 

explore-options-1.png

Share experiment results

You can share the Results page with key stakeholders using one of the following methods:

  • Click the share icon, enter their email address, and click Share.
  • Click the link icon, then copy and send the provided URL.

sharing-options-1.png

Graphs

Graphs provide a granular view of the data. The following graph types are available:

  • Improvement Over Time – Explore each variant's performance evolution and track improvements and trends across different versions.experiments-5.gif
  • Results Over Time – Track changes in experiment results over time.experiments-6.gif
  • Statistical Significance Over Time – View changes in the statistical significance of different variants over time.experiments-7.gif

Explore

The Explore tab lets you perform segmentation comparisons, funnel analysis, and other investigations for additional insights.

experiments-2.png

Segment experiment results

You can segment your results by cohorts and attributes. 

  • Segment – Segment your results by a chosen cohort of actors. experiment-segment.gif
  • Group By – Refine your results using one or more attributes.experimnets-4.gif

Add new tiles

Click + Add Tile to customize your visualization window.

  • New Visualization – Add a new exploration to the Explore tab.
  • Existing Visualization – Select an existing exploration and add it directly to the Explore tab. 
  • Filter – Add filters that you can use to narrow down data in a visualization.
  • Cohort Filter – Use cohorts to narrow down data in a visualization. 
  • Parameter – Modify the value of any placeholder parameters used in the queries of linked visualization tiles. 
  • Experiment – Add a new experiment.
  • Text – Add blocks of text anywhere in the Explore tab to provide additional context.

experiments-8.png

Adjust grid settings

Grid Settings let you alter grid configurations using the following options:

  • Grid Columns – Specify the number of columns in the grid.
  • Compact Vertically – Toggle on the compact grid view.
  • Back to default – Click to revert to default grid settings. This option becomes clickable if you make changes to the default grid settings.

Click Apply to save the changes to the grid settings.

grid-icon.png

Comments

You can add comments about items in the visualization by clicking the Comment icon, entering your notes, and clicking Send.

experiments-10.gif

To edit a comment, click More (...) > Edit Comment. Make your changes, and click Confirm to save.

To delete a comment, click More (...) > Delete Comment. Click Confirm to delete.

experiments-11.gif