Understand your Experiment Scorecard

An Experiment Scorecard template consists of the following modules:

• Measure module
• Experiment module
• Metrics module
• Segmentation module
• Filters module
• Visualization module

Measure module

The Measure module contains a selector that lets you choose a measure to analyze based on the selected event pattern and actor segments.

The following are the measure types available in the scorecard:

• Summary – Displays a scorecard for the performance of different variants across selected measures.
• Metrics over time – Lets you track metrics over time to understand how they change.
• Improvement over time – Shows how the improvement of different variants changes over time.
• Statsig over time – Displays the statistical significance of different variants over time.

Experiment details module

This module includes a selector that lets you choose an Optimizely experiment. 

When you select an experiment, the module displays related metadata for that experiment.

• Statistical significance level – The threshold at which the estimated lift on an experiment is considered statistically significant. 

  

• Baseline – The variant that is used as the point of comparison for other variants.

  

• Variations – Alternative versions of a site (in Optimizely Web Experimentation), a feature on the backend service, or a mobile app feature that are tested against the original.

Metrics module​

The metrics module lets you configure metrics for your experiment. You can use an existing metric or create a new one. There are two types of metrics:

• Decision-making metrics
• Guardrail metrics

Decision-making metrics

To create a new metric, choose one of the following options:

• Numeric aggregation – Aggregates values from existing columns in your data using a specified function.
• Conversion – Calculates the percentage of users who performed a defined set of events.
• Ratio – Creates a ratio between two metric blocks.

Numeric aggregation block​

The Numeric Aggregation block lets you create aggregations for existing columns in your data. It calculates an aggregate value for a property or block using a specified aggregation function. The output is a numerical value.

To create a numeric aggregation block, select a Measure type from the drop-down list. The following options are available:

• Conversion Rate – Percentage of actors who performed a conversion event.
• Average Event Count per Actor – Average number of events each actor performs.
• Aggregate over property of an event – Custom aggregate for actors who performed at least one of the specified events.

• Intervals Engaged – Number of time buckets where an actor met the engaged event criteria.

  

Additional configuration depends on the selected measure type.

• For Conversion Rate and Average Event Count per Actor, you must choose events as the next step.
• For Aggregate over property of an event, you must select an aggregator and set a value.
• For Intervals Engaged, configure the interval and then select the event.

You can also configure property filters within this block.​

Conversion block

A conversion block segments a dataset based on observed behaviors and associated properties. The behavior of each record in the dataset is determined by the events linked to the entity. The output of the conversion block is the percentage of users who performed the specified events.

As with the numeric aggregation block, you must choose a Measure type and configure the events for this metric type accordingly.

Ratio block

The ratio metric block lets you create a metric by dividing one metric block by another.  

You can create custom metrics by dividing the total count (total conversions), unique count (unique conversions), total revenue, or total value of one event by the total count (total conversions) or unique count (unique conversions) of another event.

Other options

The following options are available to customize your metrics:

• Rename
• Add formatting
• Add conversion window
• Add cuped duration
• Outlier management
• Guardrail alerts

Rename

To rename your metric, click More (⋮) > Rename and enter the name.

Add formatting

To add formatting to the metric configuration, enable the Formatting toggle and select a format using the drop-down list.

Add conversion window

Optimizely Experimentation calculates results by linking decision events (when a user is bucketed into a variation) with conversion events (actions the user takes, such as clicks or purchases). By default, all conversions that occur after the decision event are attributed to that variation, no matter how many days later they occur, as long as the experiment is still running.

However, with the conversion window feature, you can customize how long conversions are counted after a user is assigned to a variation (is bucketed).

For example, when creating a metric, you might define a window like the following: Only count conversions that occur within 1 day of the user being bucketed into the experiment.

This allows tighter control over what behavior is a valid conversion, focusing your analysis on the experiment's immediate impact rather than long-tail effects.

This is especially useful for actions expected to happen quickly (for instance, form submissions, clicks, and purchases) and gives you more flexibility in interpreting experiment performance.

Add cuped duration

This option changes the period of data that CUPED uses. By default, CUPED uses two weeks of historical data; however, you can change this to a custom period.

Outlier management

The scorecard presents metric results for your experiments. While displayed in the scorecard, the metrics themselves are treated as independent entities behind the scenes. In addition to these metrics, you can apply variance reduction techniques to enhance result reliability.

Outlier management helps improve the reliability and clarity of your metrics by adjusting extreme or anomalous values that could skew results and reduce metric variance. This is particularly useful for conversion metrics calculated as ratios (for example, total clicks per user or total purchase value per user).

The following are the two types of outlier management:

• Percentile – Uses the Winsorization method that lets you perform robust data analysis by adjusting outlier values in a dataset. Initially, all metric values from users are collected and represented as a range of values. A user-specified percentile, such as the 99th percentile, is then calculated to determine the range that includes the most common values, covering 99% of the data. Values exceeding this range are adjusted to match the specified percentile value, ensuring that extreme outliers do not skew the analysis while maintaining the integrity of the dataset. This process helps achieve a more accurate and reliable representation of the data, facilitating better insights and informed decision-making. 

  

• Constant – Uses the metric capping method, where the extreme values are replaced with values that are more common for the observed distribution. This lets you limit metric values using user-defined constant thresholds, rather than percentiles (as in Winsorization). It is useful when you already know the acceptable range for your data and want to force all values to stay within a fixed minimum or maximum. Setting the upper bound replaces all values greater than this constant with the upper cap.

  

For outlier management methods (percentile and constant), you can choose to apply smoothing at the Users dataset level or the Product Events dataset level.

• Users' level – Smooths outliers at the users' dataset level.

  

• Product Events level – Smooths outliers at the product events dataset level.

  

Consider the following example with a constant outlier threshold of 500 USD:

Kate and Josh are shopping on an ecommerce website and make the following transactions:

1. Kate – 200 USD
2. Kate – 600 USD
3. Josh – 800 USD

• User-level smoothing
  • Kate's total = 200 + 600 = 800 → smoothed to 500
  • Josh's total = 800 → smoothed to 500
  • Total purchase = 500 + 500 = 1,000 USD
• Product Event-level smoothing
  • Kate's purchases = 200 (ok), 600 → smoothed to 500
  • Josh's purchase = 800 → smoothed to 500
  • Total purchase = 200 + 500 + 500 = 1,200 USD

Guardrail alerts

The Set alert option lets you set thresholds on key experiment metrics and receive alerts when these thresholds are exceeded. It helps you detect negative impacts early, enabling informed decisions about whether to continue, halt, or adjust an experiment. There are two types of alert notifications: email and Slack. Alert checking occurs every six hours during the first 15 days of the experiment. After that, they occur once per day until day 30, after which no further checks are performed.

Enable guardrails

To set alerts, you must enable the Guardrails feature under Settings > General Settings > Optimizely Integration.

To add alerts,

1. Click More (⋮) > Set alert.

   

2. Set the threshold in the Alert when threshold is breached field. This setting triggers an alert when outcomes surpass or fall below a specified threshold relative to the baseline. Define a percentage change (positive or negative) and assess variations for the selected metric using relative improvement, such as a 10% to 11% change, representing a 10% relative improvement.
3. Add a visitor count in the Alert only if users amount is more than field. This triggers an alert after checking the difference between variations and the baseline for a metric, and only after the visitor count specified in this field is reached. When the visitor count is low, the metric is volatile and can fluctuate significantly. A higher number of visitors ensures that the metric value is more stable and closer to its true value. For example, you may require at least 10,000 users before an alert is sent, even if the threshold is breached earlier.
4. Choose a list of users in the Notify field.

5. Check the Alert only when statistical significance is reached option so that alerts are only triggered if the indicated threshold was breached by a variation that reached statistical significance, reducing noise from early or incomplete data. Click Save.

   

Types of alerts

You can trigger two types of alerts when experiment outcomes exceed the threshold, Slack alerts and email alerts.

Slack alerts

1. Add the Optimizely app to Slack.

   

2. Click Login to Experimentation. When you log in, you will the following Slack commands that enable you to receive notifications in different channels:

   • /subscribe – Subscribe to a particular project in a channel.
   • /unsubscribe – Unsubscribe from a particular project.
   • /unsubscribe-all – Unsubscribe from all project notifications within the channel.
   • /show-subscribed-projects – View all experimentation projects subscribed to the channel.
   • /optimizely-help – Trigger the help prompt containing help guidelines.

   

3. Open a channel of your choice and invite the Optimizely app. Type @Optimizely and hit Send.

   

4. Type /subscribe. When you subscribe to a project, all guardrail alerts that are set in any experiments of this project will display in the channel automatically.

   

5. Click the Select a project drop-down list that displays the available projects. Select a project to receive alerts.

   

The following is an example of a Slack alert:

Email alerts

Email alerts are triggered for the users you list in the Notify field. For email notifications, you can add existing Optimizely Analytics users.

Segmentation module​

The segmentation module lets you select a cohort of actors, such as users, or one or more attributes to include in the analysis. It has two subsections: Segment by Cohort and Group by Property, which let you add cohorts and attributes. You can choose to create a cohort by choosing an existing cohort from the drop-down list or by using the + New Cohort option to create a behavioral cohort block in one click.

Filters module​

You can use filters to narrow down data in a visualization. They make it easier for the user to answer exploratory questions. For example, users can define a subscription tier filter and see the narrowed-down data if they want to see results for a specific tier.

You can also choose JSON columns in this module. When you click on a JSON column, it expands to display all the available keys for that particular column. You can choose a key and click Apply. When this is done, the selected end key is chosen as the display name for that column.

Visualization module​

The visualization module in the Experiment Scorecard template lets you run and view the analysis as a pivot chart. It also lets you add the chart to a dashboard.

The following features are available in this module:

Time range

• You can configure the analysis's time range and time grain. The time range refers to the complete period during which events are considered for analysis. Examples include the last two years or the time range between two specific dates. The time range is set by default to the duration of the experiment.
• You can set the time range using a drop-down list or choose from the quick options and iterate through different choices without leaving the chart. To set a lag, click Offset and set the Ending.

Column sorting

Column sorting lets you sort the columns in the resulting pivot table.