Understand your Experiment Scorecard

  • Updated

An Experiment Scorecard template consists of the following modules:

opti-exp-4.png

Measure module

The measure module contains a selector that lets you choose a measure to analyze based on the selected event pattern and actor segments.

opti-exp-5.png

The following are the measure types available in the scorecard:

  • Summary – Scorecard for the performance of different variants across selected measures.
  • Metrics over time – View metrics over time to understand how they change.
  • Improvement over time – Explore how the improvement of different variants changes over time.
  • Statsig over time – Statistical significance of different variants over time.

Experiment details module

The experiment module contains a selector that lets you choose an Optimizely experiment. When you select an experiment, you can see other experiment metadata.

opti-exp-6.png

Statistical significance level

The threshold at which the estimated lift on an experiment is statistically significant.

opti-exp-7.png

Baseline

The variant against which other variants should be compared.

opti-exp-3.png

Variations

In Optimizely Web Experimentation, alternate versions of a site or a feature on the backend service or in the mobile app are tested against the original. 

opti-exp-8.png

Metrics module​

The metrics module lets you configure metrics for your experiment. In this section, you can use an existing metric or create one. For the new metric, you have three options: Numeric aggregation, Conversion, and Ratio.

opti-exp-9.png

Numeric aggregation block​

This block lets you create aggregations for existing columns in your data. It calculates an aggregate value for a property or block using a specified aggregation function. The output is a numerical value.opti-exp-10.png

When creating a numeric aggregation block, select a Measure type from the drop-down list. The following options are available:

  • Conversion Rate – The percentage of actors who did a conversion event.
  • Average Event Count per Actor – The average number of events each actor performs.
  • Aggregate over property of an event – Custom aggregate for actors who did at least one of the events.
  • Intervals Engaged – The count of time buckets for each actor, in which that actor met the engaged event criteria.

    opti-exp-11.png

  •  
  • When choosing the Conversion Rate and Average Event Count per Actor measures, you must choose events as the next step.
  • When choosing Aggregate over property of an event, you need to select an aggregator and set the value.
  • When choosing Intervals Engaged, you need to configure the interval and then select the event.

You can also configure property filters within this block.

Conversion block​

A conversion block segments a dataset based on observed behaviors and associated properties. The behavior of each record in the dataset is determined by the events linked to the entity. The output of the conversion block is the percentage of users who performed the events specified. As with the numeric aggregation block, you must choose a measure type and configure the events for this metric type accordingly.

opti-exp-12.png

Ratio block

The ratio metric block lets you create ratios between two metric blocks.

You can create custom metrics by dividing the total count (total conversions), unique count (unique conversions), total revenue, or total value of one event by the total count (total conversions) or unique count (unique conversions) of another event.

opti-exp-001.png

Other options

Rename

To rename your metric, click More (⋮) > Rename and enter the name.

opti-exp-13.png

Add formatting

To add formatting to the metric configuration, click More (⋮) > Add formatting.

opti-exp-14.png

Add conversion window

Optimizely Experimentation calculates results by linking decision events (when a user is bucketed into a variation) with conversion events (actions the user takes, like clicks or purchases). By default, all conversions after the decision event are attributed to that variation, no matter how many days later they occur, as long as the experiment is still running.

However, with the Conversion time window feature, you can customize how long conversions are relevant after a user is assigned to a variation.

For example, when creating a metric, you might define a window like: Only count conversions that occur within 1 day of the user being bucketed into the experiment.

This allows tighter control over what behavior is a valid conversion, focusing your analysis on the experiment's immediate impact rather than long-tail effects.

This is especially useful for actions expected to happen quickly (for instance, form submissions, clicks, purchases) and gives you more flexibility in interpreting experiment performance.

opti-exp-15.png

Add CUPED duration

This option changes the period of data that CUPED uses. CUPED uses two weeks of historical data by default, but you can change it to a custom period.

opti-exp-16.png

Add outlier management

This functionality is in beta. Contact your Customer Success Manager if interested.

The scorecard presents metric results for your experiments. The metrics themselves, while displayed in the scorecard, are treated as independent entities behind the scenes. On top of these metrics, you can apply variance reduction techniques to improve result reliability.

Outlier management helps improve the reliability and clarity of your metrics by smoothing out extreme or anomalous values that could skew results. This is particularly useful for conversion metrics calculated as ratios (for example, total clicks per user or total purchase value per user).

There are two types of outlier management:

  • Percentile – Uses the Winsorization method, where the identified outliers are replaced with values that reflect the average within the retained range, ensuring continuity and comparability without distortion. You can define a percentile threshold that determines which values are considered outliers. The percentile range is 90 – 99.9. A 99.9 percentile upper bound means, "Adjust values higher than 99.9% of all other values." 

    opti-exp-17.png

  • Constant – Uses the metric capping method, where the extreme values are replaced with values that are more common for the observed distribution. This lets you limit metric values using user-defined constant thresholds, rather than percentiles (as in Winsorization). It is useful when you already know the acceptable range for your data and want to force all values to stay within a fixed minimum or maximum. Setting the upper bound replaces all values greater than this constant with the upper cap.

    opti-exp-18.png

For both outlier management methods (percentile and constant), you can choose to apply smoothing at the actor level or the event level.

  • Actor level – Smooths outliers based on the total value per actor (for example, a visitor).

    outmgmt-actor.png

  • Event level – Smooths each individual event separately.

    outmgmt-events.png

Consider the following example with a constant outlier threshold of 500 USD.

Kate and Josh are shopping on an ecommerce website and make the following transactions:

  1. Kate – 200 USD
  2. Kate – 600 USD
  3. Josh – 800 USD
  • Actor-level smoothing
    • Kate's total = 200 + 600 = 800 → smoothed to 500
    • Josh's total = 800 → smoothed to 500
    • Total purchase = 500 + 500 = 1,000 USD
  • Event-level smoothing
    • Kate's purchases = 200 (ok), 600 → smoothed to 500
    • Josh's purchase = 800 → smoothed to 500
    • Total purchase = 200 + 500 + 500 = 1,200 USD

Segmentation module​

The segmentation module lets you choose a cohort of actors, such as users, or one or more attributes to include in the analysis. It has two sub-sections: Segment by Cohort and Group by Property, which let you add cohorts and attributes. You can choose to create a cohort by choosing an existing cohort from the drop-down list or by using the + New Cohort option to create a behavioral cohort block in one click.

opti-exp-19.png

Filters module​

You can use filters to narrow down data in a visualization. They make it easier for the user to answer exploratory questions. For example, if the user wants to see results for a specific tier, they can define a subscription tier filter and see the narrowed-down data.

Analytics also lets you choose JSON columns in this module. When you click on a JSON column, it expands to display all the available keys for that particular column. You can choose a key and click Apply. When this is done, the selected end key is chosen as the display name for that column.

opti-exp-20.png

Visualization module​

The visualization module in the Experiment Scorecard template enables you to run and view the analysis as a pivot chart. It also provides the ability to add the chart to a dashboard.

opti-exp-21.png

The following features are available in this module.

Time Range

  • You can configure the analysis's time range and time grain. Time Range refers to the complete period during which events are considered for the analysis. Examples include the last 2 years or the time range between two specific dates. The time range is set by default to the duration of the experiment.
  • You can set the time range using a drop-down list or even choose from the quick options and iterate through different choices without leaving the chart. It is also possible to set a lag by clicking Offset and setting the Ending.

opti-exp-22.png

Column sorting

This section lets you sort the columns in the resulting pivot table.

opti-exp-23.png