New A/B Results page

The A/B results page builds on the capabilities of the classic results page and introduces advanced analytical features — including CUPED, configurable stats engines, cohort segmentation, and deeper data health checks — to give Web Experimentation and Feature Experimentation customers a more powerful way to measure and interpret A/B test performance.

Review each metric and variation to see how visitors respond to your site or application changes. Use segments to investigate visitor behavior in more detail.

The Stats Engine in Optimizely Experimentation powers the experiment results page. It provides a data-rich view of visitor interactions, includes confidence intervals, and applies false discovery rate control based on the selected statistical method — Sequential, Frequentist (Fixed Horizon) and Bayesian statistics.

The following details are important to know:

• Visitor bucketing – Visitors are randomly bucketed into variations based on your traffic distribution settings, so variation audiences may differ in size.
• Local time and data freshness – Results display in your computer’s time zone and typically become available within five to ten minutes of data ingestion. See Data freshness.

Make sure the following are in place before you access your experiment results:

• An active or completed experiment — You need a running or completed A/B experiment with collected visitor data. Results are typically available within five to ten minutes of data ingestion.
• Project access with Viewer role or higher — You need at least viewer-level permissions on the project to view experiment results. See Manage collaborators for details on roles and permissions.

Access experiment results

To access the Optimizely results page:

• Optimizely Web Experimentation – Go to Optimizations > Overview > Results > Real-time Results (New Version). 

  

• Optimizely Feature Experimentation – Go to Reports and select a report or go to Flags and select a flag. This would display the results for a specific rule within the flag. 

  

The results page has a Summary and Explore tab. Learn how to access Experiments.

The Summary tab has key insights and gives you an overview of performance of all of your experiment variations against your metrics from the selected experiment to support decision-making. 

The Explore tab has two functions:

• It lets you explore data and get deeper insights using features like segmentation and filtering.
• It lets you create a compelling narrative for experiment results using previously created explorations and adding explanatory comments.

Summary

The Summary tab provides an overview of the experiment information configured during experiment setup in Optimizely, including the following:

• Name – Shows the name of the experiment.
• Status – Shows the experiment type (A/B test) and the current live status. See the difference between publish, start, and pause.
• Visitors – Shows a high-level snapshot of total traffic exposure and conversion volume across all variations. See Target visitors with audience conditions.
• Project – Shows the Optimizely project this experiment belongs to.
• Environment (Feature experimentation only)– Shows the environment the experiment is running in.

• Audiences – Lists audiences targeted in the A/B test.

  

• Experiment health and Sample Ratio Mismatch (SRM) detection – Indicates experiment health. Unlike the legacy results page, SRM is not triggered automatically and you have to manually trigger it to see the result. See automatic experiment health indicator.

  

• Results last updated and last event – Shows when results were last updated and the timestamp of the most recent event based on your computer's time zone.

• Days running – Shows the total full days the A/B test has run. Optimizely Experimentation truncates any floating-point number part of a day. For example, 17.8 displays as 17.

  

Under Advanced Settings, you have four configurable controls:

• Baseline – Compares the results for all variations against the variation you choose here. You can change this to any variation using the drop-down list.
• CUPED – Reduces metric variance using pre-experiment data to help reach significance faster.
• Stats engine – Signifies that the experiment uses Optimizely's sequential Stats Engine, which lets you peek at results at any time without inflating false positive rates. You can configure the stats engine when you set up your experiment and you have the following options: Sequential (default), Frequentist (Fixed Horizon), and Bayesian. The stats engine that you select displays on the results page.

• Statistical significance threshold – Sets the significance threshold against which results are evaluated. You can configure the threshold when you set up your experiment. Results are considered statistically significant when they reach or exceed this level.

  

Edit experiment

Change experiment settings or view the experiment within Optimizely. Select the external link icon to return to your experiment setup. 

Manage metrics

The results page lets you sync metrics from the experiment setup. Unlike the classic results page, you can add decision-making or guardrail metrics and remove or edit previously added metrics directly within the results page.

Each metric has the following advanced options:

Add conversion window

Optimizely Experimentation calculates results by linking decision events (when a user is bucketed into a variation) with conversion events (actions the user takes, such as clicks or purchases). By default, Optimizely attributes every conversion that follows the decision event to that variation, regardless of the delay, for as long as the experiment runs.

The conversion window lets you set how long Optimizely counts conversions after assigning a user to a variation (also called bucketing).

For example, when you create a metric, you can define a window such as: Count only the conversions that occur within one day of bucketing.

A conversion window gives you tighter control over what counts as a valid conversion and focuses analysis on the immediate impact of the experiment rather than long-tail effects.

Conversion windows are especially useful for actions that happen quickly, such as form submissions, clicks, and purchases. They also give you more flexibility when you interpret experiment performance.

Add CUPED duration

The CUPED duration sets the historical data period that CUPED (Controlled-experiment Using Pre-Experiment Data) uses to reduce variance in your results, helping you detect smaller effects faster. By default, CUPED uses two weeks of historical data. To use a custom period, adjust the duration.

Outlier management

Outliers are unusually high or low values that can distort your results. Outlier management adjusts these values so your data better represents typical behavior.

The scorecard presents metric results for your experiments. Each metric is treated as an independent entity. Apply variance reduction techniques to enhance result reliability.

Outlier management improves the reliability of your metrics by adjusting extreme or anomalous values that would otherwise skew results, and it reduces metric variance. Outlier management is especially useful for conversion metrics that are calculated as ratios, such as total clicks per user or total purchase value per user.

The following are the two types of outlier management:

• Percentile – Uses Winsorization to adjust outlier values in your dataset. Winsorization replaces the most extreme values in the data with the value at a chosen percentile, so a few unusual points cannot dominate the average.
  First, Optimizely collects the metric values from every user and represents them as a range. Optimizely then calculates a percentile you specify, for example the 99th percentile, to define the range that covers the most common values. Optimizely adjusts every value outside this range to the percentile value you specified, so extreme outliers do not skew the analysis.

  

• Constant – Uses metric capping. Optimizely replaces extreme values with a constant you define, keeping every metric value inside a fixed minimum or maximum range.
  The Constant method limits metric values with thresholds you define, rather than the percentile thresholds that Winsorization uses. Use the Constant method when you already know the acceptable range for your data and want every value to stay inside a fixed minimum or maximum. Setting the upper bound replaces every value above the constant with the cap you set.

  

Apply smoothing to either the Users dataset or the Product Events dataset, for either the Percentile or Constant method.

• Users level – Smooths outliers at the users dataset level.

  

• Product Events level – Smooths outliers at the product events dataset level.

  

The following example uses a Constant (bold) outlier threshold of $500 USD:

Kate and Josh are shopping on an ecommerce website and make the following transactions:

1. Kate – 200 USD
2. Kate – 600 USD
3. Josh – 800 USD

User-level smoothing

• Kate's total = $200 + $600 = $800 → capped at $500
• Josh's total = $800 → capped at $500
• Total = $500 + $500 = $1,000

Product Event-level smoothing

• Kate's purchases = $200 (no change), $600 → capped at $500
• Josh's purchase = $800 → capped at $500
• Total = $200 + $500 + $500 = $1,200

Guardrail alerts

The Set alerts option lets you set thresholds on key experiment metrics and receive alerts when a metric crosses one. Alerts help you detect negative impacts early so you can decide whether to continue, halt, or adjust an experiment. There are two types of alert notifications: email and Slack. Optimizely checks alerts every six hours for the first 15 days of the experiment, then once a day until day 30. After day 30, checks stop.

Enable guardrails

Before you set alerts, go to Settings > General Settings > Optimizely Integration and enable Guardrails.

To add an alerts:

1. Toggle Set alerts.

   

2. Set the threshold in the Alert when threshold is breached field. The alert triggers when a variation's outcome rises above or falls below the threshold you set, relative to the baseline. Define a percentage change (positive or negative) and assess variations for the selected metric using relative improvement, for example, a baseline of 10% rising to 11% is a 10% relative improvement.
3. Enter a visitor count in the Alert only if users count is at least field. The alert triggers only after the visitor count in this field is reached, and only after Optimizely has measured the difference between each variation and the baseline. When the visitor count is low, the metric is volatile and fluctuates widely. A higher visitor count stabilizes the metric value and brings it closer to its true value. For example, you may require at least 10,000 users before Optimizely sends an alert, even when a variation breaches the threshold earlier.
4. Choose the users to notify in the Notify field.
5. Check the Alert only when Statsig is reached option to trigger alerts only when a variation reaches statistical significance and breaches the threshold. This setting reduces noise from early or incomplete data.

6. Click Save.

   

Types of alerts

Optimizely sends two types of alerts when a variation crosses the threshold: Slack and email.

Slack alerts

1. Add the Optimizely app to Slack.

   

2. Click Login to Experimentation. After you log in, the following Slack commands let you receive notifications in different channels:

   • /subscribe – Subscribe to a project in a channel.
   • /unsubscribe – Unsubscribe from a project.
   • /unsubscribe-all – Unsubscribe from all project notifications within the channel.
   • /show-subscribed-projects – View all experimentation projects subscribed to the channel.
   • /optimizely-help – Open the help prompt containing help guidelines.

   

3. Open a channel of your choice and invite the Optimizely app. Type @Optimizely and click Send.

   

4. Type /subscribe. After you subscribe to a project, every guardrail alert set in any experiment in that project displays in the channel automatically.

   

5. Click the Select a project drop-down list to see the available projects. Select a project to receive alerts.

   

The following screenshot shows an example Slack alert:

Email alerts

Email alerts go to the users you list in the Notify field. Add existing Optimizely Analytics users to receive email notifications

The following screenshot shows an example email alert:

Sample Ratio Mismatch (SRM) detection

A sample ratio mismatch (SRM) occurs when users are unexpectedly imbalanced across your experiment's variations. An imbalance can signal issues with your experiment configuration or external factors, which may invalidate your results. Learn about sample ratio mismatch (SRM) detection.

Click Check SRM status in the Experiments section to see the latest health status of your experiment traffic distribution.

Health check overview

Check health verifies the integrity of data used in experiments to ensure accurate results and reliable decision-making. Check health runs the following verification:

• Dataset primary key uniqueness – Runs a primary key check on the actor dataset to verify that each actor identifier is unique. Learn about the primary key health check.
• Actor identifier alignment – Compares actor identifiers across the event, decision, and actor datasets. Significant misalignment usually means a wrong column was selected during experiment configuration, or there is a broader data-integrity issue to investigate.
• Single variation per actor – Counts actors that were assigned conflicting variations and excludes them from the analysis. A high count usually indicates the experiment is misconfigured.

Each check returns one of four health statuses. The status determines which data configuration you need to adjust:

• Healthy – The data passed the check.
• Unhealthy – The check found a critical data-integrity issue.
• Warning – The check detected a potential issue that could affect the accuracy of your results.
• Skipped – The check did not run because the primary key configuration is invalid (the selected columns are incompatible or misconfigured).

Share experiment results

Share the Results page with key stakeholders by clicking Share. The Share Results dialog displays the link to the results page. Click Copy Link to copy and send the provided URL. Click Reset Link to reset the URL. 

Graphs

Graphs provide a granular view of the data. The following graph types are available:

• Improvement over time (default) – Improvement in this metric for each variation compared to the baseline. See Confidence intervals and improvement intervals.
• Metric over time – Conversions per day in this metric for each variation, including the original.
• Statistical significance over time – Cumulative statistical significance for the variation. 

Explore

The Explore tab lets you perform segmentation comparisons, funnel analysis, and other explorations for additional insights.

There are three ways to access the Explore tab: 

• Click Explore in the experiment results page.

  

• Click one of the following exploration options within the Summary tab: Change Date Range, Segment by Cohorts, Group By Property, and Filter out data.

  • Change Date Range - Narrow or expand the results view to a specific time window within the experiment's runtime.
  • Segment by Cohorts – Filter results to a defined group of actors to understand how a specific cohort responds to each variation. 
  • Group By Property – Break down results by one or more attributes (for example, device type or browser) to identify performance differences across audience segments.
  • Filter out data – Exclude specific data points or conditions from the results view to focus analysis on relevant traffic.

  

• Click Explore results data on a metric within the Summary tab to open the Explore tab and dive deeper into that metric's data.

  

Exploration summary with Optimizely Opal

Prerequisites

• You must use Opti ID to access Opal.
• Your Optimizely Web and Feature experimentation instances must be enabled for Opti ID.
• You must have generative AI enabled in Optimizely.

Optimizely Opal interprets and summarizes data in the results module so you do not need to scan result tables manually. Select the summarize icon in the visualization window to summarize your exploration. 

The chat displays the following information:

• A brief summary of your experiment results
• Key takeaways
• Next steps and suggestions

Segment experiment results

Segment results by cohorts and attributes. 

• Segment – Click Segment and segment your results by a chosen cohort of actors. Filter results based on a dynamically defined behavioral cohort, such as visitors who performed a specific sequence of actions during the experiment (for example, visitors who did action A and then action B). Click Add to Explore to add it to the result analysis.

  

• Group By – Refine your results using one or more attributes. When you group results by attributes, it creates a results breakdown based on the selected property. For example, you can compare the results for desktop versus mobile users, or users in different countries. Click Add to Explore to add it to the result analysis.

  

Add tiles

Click + Add Tile to customize your visualization window.

• New Visualization – Add a new exploration to the Explore tab.
• Existing Visualization – Select an existing exploration and add it directly to the Explore tab. 
• Filter – Add filters that you can use to narrow down data in a visualization.
• Parameter – Modify the value of any placeholder parameters used in the queries of linked visualization tiles. 
• Text – Add blocks of text anywhere in the Explore tab to provide additional context.

Visualization options 

The visualization module interprets the results of the metrics defined in the Summary tab. It displays a summary first, followed by individual metric interpretations. Select Explore results data on any metric to dive deeper into that metric's data in the Explore tab.

The Summary section provides a high-level view of how each variation performs across all metrics. For each variation, it displays:

• Visitors – The total number of visitors and their percentage share of experiment traffic.
• Metric value – The raw metric value for each variation across all configured metrics.

• Improvement – The relative percentage change compared to the baseline (Original). The Original row displays – as it is the comparison point.

  

The Primary label identifies which metric the stats engine prioritizes for significance.

Following the summary, each metric is listed individually and numbered by priority. The primary metric is ranked 1 and labeled Primary. For each metric, the following columns are displayed:

• Metric value – The raw computed value of the metric for each variation.
• Unique conversion – The numerator (unique conversions) and denominator (total visitors) used to calculate the metric value.
• Improvement – The relative percentage change over the baseline. Displays – for the Original and a positive or negative percentage for each variation.
• Confidence interval – A visual bar representing the range within which the true improvement is estimated to fall. When the interval sits entirely above or below zero, the result is statistically significant.

• Stat Sig Level – The statistical significance level reached by the variation for that metric.

  

Each metric includes an Analyze Over Time graph with three toggleable views:

• Improvement (Default) – Tracks how the relative lift of each variation evolves over the experiment duration.
• Metric value – Shows how the raw metric value changes over time for each variation.

• Statistical significance – Displays how the significance level trends as data accumulates.

  

Comments

Add comments about items in the visualization by clicking Comment, entering your notes, and clicking Send.

To edit a comment, click More (...) > Edit Comment. Make your changes, and click Confirm to save.

To delete a comment, click More (...) > Delete Comment. Click Confirm to delete.

Status

Select Status to update the current state of the experiment using the drop-down list. The available options are:

• Running – Keeps the experiment live and actively collecting data.
• Pause – Temporarily stops the experiment without archiving it.
• Conclude – Ends the experiment and locks the results.
• Archive – Removes the experiment from the active view.

Download results

Click Download to export the current results as a CSV file. The export reflects the data displayed on the page, including any active filters, segments, or date range selections.

Other options

Click More (⋮) to access two additional actions:

• Switch to classic view – Returns to the classic experiment results page.
• Reset results – Clears the current results data and returns counts to zero. Note that resetting results does not affect variation bucketing — visitors continue to see the same variation they were previously assigned.