Experiment analysis

  • Updated

In Optimizely, experiments are structured tests designed to compare different variations of a webpage, feature, or experience to determine which performs best based on specific goals. These experiments help businesses optimize user experiences, increase conversions, and make data-driven decisions. All experiments created within Optimizely are visible on the Experiments page in Optimizely Warehouse-Native Experimentation Analytics.

Optimizely Analytics currently only supports A/B tests and multivariate tests as experiment types.

Optimizely offers different types of experiments for various objectives. Some experiments focus on learning and long-term insights, such as A/B tests, multivariate tests, and Stats Accelerator, which incorporate statistical significance calculations. Others prioritize immediate impact, like a multi-armed bandit (MAB) and the contextual bandit, which optimize for short-term gains without performing statistical significance analysis. Learn more about distribution modes and experiment types.

Access experiments

Go to Experiments to see a quick overview of the latest experiments. 

You can refine the list of experiments by adjusting the filters (such as date range) or selecting one or more Types.

Click any experiment to edit the associated scorecards. 

Experiment results

Each experiment scorecard has a Summary and Explore tab.

The Summary tab has key insights from the selected experiment to support decision-making.

The Explore tab lets you further analyze data within the experiment and its variations.

The following options are available on the Explore tab: 

explore-options-bottom.png

Summary tab

The Summary tab overviews the exploration, including the selected experiment, configured decision-making metrics, and the experiment results in the visualization window.

You can use graphs, date ranges, attributes, and the baseline to determine results. You can modify the scorecard configuration within the tab. Learn more about creating scorecards.

The experiment details block in the summary shows the audiences for the experiment and, for concluded experiments, the conclusion.

  • The Audiences field shows the audience conditions for the experiment as a combined expression, for example, Returning visitors and (Mobile or Tablet). When no conditions are set, the field shows Everyone. Analytics truncates long expressions and shows the full set in a tooltip.

  • For concluded experiments, the block also shows Deployed variation, Results outcome, and Conclusions, so the decision and its rationale stay visible on the results page.

Click Hide Editor to collapse the editor pane and free up horizontal space when you review results. Click Expand Editor to display it again.

Edit experiment

You can change experiment settings and see the experiment within Optimizely. Click the external link icon to go to the Flags section in Feature Experimentation.  

Change the baseline

The Baseline option lets you compare your variations against a specific one instead of the original. To do so, select your preferred variation from the Baseline drop-down list.

Manage metrics

You can add a new primary or guardrail metric and remove or edit previously added metrics. When you add a metric to the scorecard, choose from the following types, which match the coverage available in explorations:

adding-metrics.png
  • Conversion – Create event segmentation style aggregations, such as count of events or count of users.
  • Numeric aggregation – Create aggregations over existing columns in your data.
  • Ratio – Create a ratio between two metrics.
  • Funnel – Measure full-funnel conversion rate through a sequence of steps.
  • Bounce – Measure visitor engagement when visitors first interact with your website.
  • Exit – Measure visitor engagement on pages further down in your conversion funnel.

To reuse a scorecard measure elsewhere, click More (⋮) > Save as Metric. Enter a Name, Description, select the Folder and click Save.

Visualization options

Within Summary > Visualization, you can select segmentation options (Segment and Group By) and graph options (Improvement Over Time, Results Over Time, and Statistical Significance Over Time). You can toggle between different graphs for each metric. 

exploreoptions.png

Sample Ratio Mismatch (SRM) detection

A sample ratio mismatch (SRM) occurs when users are unexpectedly imbalanced across your experiment's variations. An imbalance can signal issues with your experiment configuration or external factors, which may invalidate your results. Learn about sample ratio mismatch (SRM) detection.

Click Check SRM status in the Experiments section to see the latest health status of your experiment traffic distribution.

srm-checkhealth.png
Unlike in Optimizely Web and Feature Experimentation, where the SRM status updates automatically on page load, the status in Optimizely Analytics requires a manual refresh. It is important to note that triggering this update does not run a new, real-time analysis. Instead, it retrieves and displays the most recent status from the automated checks that run periodically in the background. 

Health check overview

Check health verifies the integrity of data used in experiments to ensure accurate results and reliable decision-making. Check health runs the following verification:

  • Dataset primary key uniqueness – Runs a primary key check on the actor dataset to verify that each actor identifier is unique. Learn about the primary key health check.
  • Actor identifier alignment – Compares actor identifiers across the event, decision, and actor datasets. Significant misalignment usually means a wrong column was selected during experiment configuration, or there is a broader data-integrity issue to investigate.
  • Single variation per actor – Counts actors that were assigned conflicting variations and excludes them from the analysis. A high count usually indicates the experiment is misconfigured.

Each check returns one of four health statuses. The status determines which data configuration you need to adjust:

  • Healthy – The data passed the check.
  • Unhealthy – The check found a critical data-integrity issue.
  • Warning – The check detected a potential issue that could affect the accuracy of your results.
  • Skipped – The check did not run because the primary key configuration is invalid (the selected columns are incompatible or misconfigured).
exp-health-2.png

Share experiment results

You can share the Results page with key stakeholders using one of the following methods:

  • Click the share icon, enter their email address, and click Share.
  • Click the link icon, then copy and send the provided URL.
shareexp.png

Graphs

Graphs provide a granular view of the data. The following graph types are available:

  • Improvement Over Time – Explore each variant's performance evolution and track improvements and trends across different versions.

    impovertime.png

  • Results Over Time – Track changes in experiment results over time.

    resultovertime.png

  • Statistical Significance Over Time – View changes in the statistical significance of different variants over time.

    statsigovertime.png

Explore tab

The Explore tab lets you perform segmentation comparisons, funnel analysis, and other investigations for additional insights.

experiments-2.png

Exploration summary with Optimizely Opal

Prerequisites

  • You must use Opti ID to access Opal.
  • Your Optimizely Analytics instance must be enabled for Opti ID.
  • You must have generative AI enabled in Optimizely.

If you use Opti ID, administrators can turn off generative AI in the Opti ID Admin Center. See Turn generative AI off across Optimizely applications.

Using Optimizely Opal, you can interpret and summarize the data presented in your explorations without having to scan through visualizations and tables manually. Click the summarize icon in the visualization window to summarize your exploration.

Summaries are not available for cohorts, metrics, dashboards, or other entities in Analytics.

exploration-summary-1.png

The chat displays the following information:

  • A brief summary of your exploration
  • Key takeaways
  • Next steps and suggestions
exploration-summary-2.png

Segment experiment results

You can segment your results by cohorts and attributes. 

  • Segment – Segment your results by a chosen cohort of actors. 

    segbycohort.png

  • Group By – Refine your results using one or more attributes.

    grpbyprop.png

Add new tiles

Click + Add Tile to customize your visualization window.

  • New Visualization – Add a new exploration to the Explore tab.
  • Existing Visualization – Select an existing exploration and add it directly to the Explore tab. 
  • Filter – Add filters that you can use to narrow down data in a visualization.
  • Cohort Filter – Use cohorts to narrow down data in a visualization. 
  • Parameter – Modify the value of any placeholder parameters used in the queries of linked visualization tiles. 
  • Experiment – Add a new experiment.
  • Text – Add blocks of text anywhere in the Explore tab to provide additional context.
experiments-8.png

Adjust grid settings

Grid Settings let you alter grid configurations using the following options:

  • Grid Columns – Specify the number of columns in the grid.
  • Compact Vertically – Toggle on the compact grid view.
  • Back to default – Click to revert to default grid settings. This option becomes clickable if you make changes to the default grid settings.

Click Apply to save the changes to the grid settings.

grid-icon.png

Comments

You can add comments about items in the visualization by clicking the Comment icon, entering your notes, and clicking Send.

opti-es-16.png

To edit a comment, click More (...) > Edit Comment. Make your changes, and click Confirm to save.

To delete a comment, click More (...) > Delete Comment. Click Confirm to delete.

opti-es-17.png