Understand your Experiment Scorecard

An Experiment Scorecard template consists of the following modules:

• Measure module
• Experiment module
• Metrics module
• Segmentation module
• Filters module
• Visualization module

Measure module

The Measure module contains a selector that lets you choose a measure to analyze based on the selected event pattern and actor segments.

The following are the measure types available in the scorecard:

• Summary – Displays a scorecard for the performance of different variants across selected measures.
• Metrics over time – Lets you track metrics over time to understand how they change.
• Improvement over time – Shows how the improvement of different variants changes over time.
• Statsig over time – Displays the statistical significance of different variants over time.

Experiment details module

This module includes a selector that lets you choose an Optimizely experiment. 

When you select an experiment, the module displays related metadata for that experiment.

• Statistical significance level – The threshold at which the estimated lift on an experiment is considered statistically significant. 

  

• Baseline – The variant that is used as the point of comparison for other variants.

  

• Variations – Alternative versions of a site (in Optimizely Web Experimentation), a feature on the backend service, or a mobile app feature that are tested against the original.

Metrics module​

The metrics module lets you configure metrics for your experiment. You can use an existing metric or create a new one. There are two types of metrics:

• Decision-making metrics
• Guardrail metrics

Decision-making metrics

To create a new metric, choose one of the following options:

• Numeric aggregation – Aggregates values from existing columns in your data using a specified function.
• Conversion – Calculates the percentage of users who performed a defined set of events.
• Ratio – Creates a ratio between two metric blocks.

Numeric aggregation block​

The Numeric Aggregation block lets you create aggregations for existing columns in your data. It calculates an aggregate value for a property or block using a specified aggregation function. The output is a numerical value.

To create a numeric aggregation block, select a Measure type from the drop-down list. The following options are available:

• Conversion Rate – Percentage of actors who performed a conversion event.
• Average Event Count per Actor – Average number of events each actor performs.
• Aggregate over property of an event – Custom aggregate for actors who performed at least one of the specified events.

• Intervals Engaged – Number of time buckets where an actor met the engaged event criteria.

  

Additional configuration depends on the selected measure type.

• For Conversion Rate and Average Event Count per Actor, you must choose events as the next step.
• For Aggregate over property of an event, you must select an aggregator and set a value.
• For Intervals Engaged, configure the interval and then select the event.

You can also configure property filters within this block.​

Conversion block

A conversion block segments a dataset based on observed behaviors and associated properties. The behavior of each record in the dataset is determined by the events linked to the entity. The output of the conversion block is the percentage of users who performed the specified events.

As with the numeric aggregation block, you must choose a Measure type and configure the events for this metric type accordingly.

Ratio block

The ratio metric block lets you create a metric by dividing one metric block by another.  

You can create custom metrics by dividing the total count (total conversions), unique count (unique conversions), total revenue, or total value of one event by the total count (total conversions) or unique count (unique conversions) of another event.

Other options

The following options are available to customize your metrics:

• Rename
• Add formatting
• Add conversion window
• Add cuped duration
• Outlier management
• Guardrail alerts

Rename

Click More (⋮) > Rename. Enter the name and press Enter.

Add formatting

Format the metric value (for example, as a percentage or decimal) so the scorecard displays it in the units of your choosing. To apply a format, click More (⋮) > Add formatting, and select a format from the drop-down list.

Add conversion window

Optimizely Experimentation calculates results by linking decision events (when a user is bucketed into a variation) with conversion events (actions the user takes, such as clicks or purchases). By default, Optimizely attributes every conversion that follows the decision event to that variation, regardless of the delay, for as long as the experiment runs.

The conversion window lets you set how long Optimizely counts conversions after assigning a user to a variation (also called bucketing).

For example, when you create a metric, you can define a window such as: Count only the conversions that occur within one day of bucketing.

A conversion window gives you tighter control over what counts as a valid conversion and focuses analysis on the immediate impact of the experiment rather than long-tail effects.

Conversion windows are especially useful for actions that happen quickly, such as form submissions, clicks, and purchases. They also give you more flexibility when you interpret experiment performance.

Add CUPED duration

The CUPED duration sets the historical data period that CUPED (Controlled-experiment Using Pre-Experiment Data) uses to reduce variance in your results, helping you detect smaller effects faster. By default, CUPED uses two weeks of historical data. To use a custom period, adjust the duration.

Outlier management

Outliers are unusually high or low values that can distort your results. Outlier management adjusts these values so your data better represents typical behavior.

The scorecard presents metric results for your experiments. Each metric is treated as an independent entity. Apply variance reduction techniques to enhance result reliability.

Outlier management improves the reliability of your metrics by adjusting extreme or anomalous values that would otherwise skew results, and it reduces metric variance. Outlier management is especially useful for conversion metrics that are calculated as ratios, such as total clicks per user or total purchase value per user.

The following are the two types of outlier management:

• Percentile – Uses Winsorization to adjust outlier values in your dataset. Winsorization replaces the most extreme values in the data with the value at a chosen percentile, so a few unusual points cannot dominate the average.
  First, Optimizely collects the metric values from every user and represents them as a range. Optimizely then calculates a percentile you specify, for example the 99th percentile, to define the range that covers the most common values. Optimizely adjusts every value outside this range to the percentile value you specified, so extreme outliers do not skew the analysis.

  

• Constant – Uses metric capping. Optimizely replaces extreme values with a constant you define, keeping every metric value inside a fixed minimum or maximum range.
  The Constant method limits metric values with thresholds you define, rather than the percentile thresholds that Winsorization uses. Use the Constant method when you already know the acceptable range for your data and want every value to stay inside a fixed minimum or maximum. Setting the upper bound replaces every value above the constant with the cap you set.

  

Apply smoothing to either the Users dataset or the Product Events dataset, for either the Percentile or Constant method.

• Users level – Smooths outliers at the users dataset level.

  

• Product Events level – Smooths outliers at the product events dataset level.

  

The following example uses a Constant (bold) outlier threshold of $500 USD:

Kate and Josh are shopping on an ecommerce website and make the following transactions:

1. Kate – 200 USD
2. Kate – 600 USD
3. Josh – 800 USD

User-level smoothing

• Kate's total = $200 + $600 = $800 → capped at $500
• Josh's total = $800 → capped at $500
• Total = $500 + $500 = $1,000

Product Event-level smoothing

• Kate's purchases = $200 (no change), $600 → capped at $500
• Josh's purchase = $800 → capped at $500
• Total = $200 + $500 + $500 = $1,200

Guardrail alerts

The Set alerts option lets you set thresholds on key experiment metrics and receive alerts when a metric crosses one. Alerts help you detect negative impacts early so you can decide whether to continue, halt, or adjust an experiment. There are two types of alert notifications: email and Slack. Optimizely checks alerts every six hours for the first 15 days of the experiment, then once a day until day 30. After day 30, checks stop.

Enable guardrails

Before you set alerts, go to Settings > General Settings > Optimizely Integration and enable Guardrails.

To add an alerts:

1. Toggle Set alerts.

   

2. Set the threshold in the Alert when threshold is breached field. The alert triggers when a variation's outcome rises above or falls below the threshold you set, relative to the baseline. Define a percentage change (positive or negative) and assess variations for the selected metric using relative improvement, for example, a baseline of 10% rising to 11% is a 10% relative improvement.
3. Enter a visitor count in the Alert only if users count is at least field. The alert triggers only after the visitor count in this field is reached, and only after Optimizely has measured the difference between each variation and the baseline. When the visitor count is low, the metric is volatile and fluctuates widely. A higher visitor count stabilizes the metric value and brings it closer to its true value. For example, you may require at least 10,000 users before Optimizely sends an alert, even when a variation breaches the threshold earlier.
4. Choose the users to notify in the Notify field.
5. Check the Alert only when Statsig is reached option to trigger alerts only when a variation reaches statistical significance and breaches the threshold. This setting reduces noise from early or incomplete data.

6. Click Save.

   

Types of alerts

Optimizely sends two types of alerts when a variation crosses the threshold: Slack and email.

Slack alerts

1. Add the Optimizely app to Slack.

   

2. Click Login to Experimentation. After you log in, the following Slack commands let you receive notifications in different channels:

   • /subscribe – Subscribe to a project in a channel.
   • /unsubscribe – Unsubscribe from a project.
   • /unsubscribe-all – Unsubscribe from all project notifications within the channel.
   • /show-subscribed-projects – View all experimentation projects subscribed to the channel.
   • /optimizely-help – Open the help prompt containing help guidelines.

   

3. Open a channel of your choice and invite the Optimizely app. Type @Optimizely and click Send.

   

4. Type /subscribe. After you subscribe to a project, every guardrail alert set in any experiment in that project displays in the channel automatically.

   

5. Click the Select a project drop-down list to see the available projects. Select a project to receive alerts.

   

The following screenshot shows an example Slack alert:

Email alerts

Email alerts go to the users you list in the Notify field. Add existing Optimizely Analytics users to receive email notifications

The following screenshot shows an example email alert:

Segmentation module​

The segmentation module lets you select a cohort of actors, such as users, or one or more attributes to include in the analysis. It has two subsections: Segment by Cohort and Group by Property, which let you add cohorts and attributes. You can choose to create a cohort by choosing an existing cohort from the drop-down list or by using the + New Cohort option to create a behavioral cohort block in one click.

Filters module​

You can use filters to narrow down data in a visualization. They make it easier for the user to answer exploratory questions. For example, users can define a subscription tier filter and see the narrowed-down data if they want to see results for a specific tier.

You can also choose JSON columns in this module. When you click on a JSON column, it expands to display all the available keys for that particular column. You can choose a key and click Apply. When this is done, the selected end key is chosen as the display name for that column.

Visualization module​

The visualization module in the Experiment Scorecard template lets you run and view the analysis as a pivot chart. It also lets you add the chart to a dashboard.

The following features are available in this module:

Time range

• You can configure the analysis's time range and time grain. The time range refers to the complete period during which events are considered for analysis. Examples include the last two years or the time range between two specific dates. The time range is set by default to the duration of the experiment.
• You can set the time range using a drop-down list or choose from the quick options and iterate through different choices without leaving the chart. To set a lag, click Offset and set the Ending.

Column sorting

Column sorting lets you sort the columns in the resulting pivot table.