Understanding your Experiment Scorecard

Template Structure​

An Experiment Scorecard template consists of the following modules:

• Measure module
• Experiment module
• Metrics module
• Segmentation module
• Filters module
• Visualization module

Let's see how each module is configured in the template.

Measure Module

The Measure module in the Experiment Scorecard template contains a selector that lets you choose a measure to analyze based on the selected event pattern and actors segments.

Running a Funnel analysis helps you measure the following:

Experiment Module​

• The Experiment module in the template contains a selector that lets you choose an Optimizely experiment. When you select an experiment, you can see other experiment metadata.

• Threshold - The threshold at which the estimated lift on an experiment is determined to be statistically signifncant.

• Baseline is the variant against which other variants should be compared.

Metrics Module​

The Metrics module in the Experiment Scorecard template allows you to configure metrics for your experiment. In this section, you can either choose to use a metric that already exists in Analytics or you can create a new metric. For the new metric, you have two options: Numeric aggregation, Conversion, and Ratio.

Numeric aggregation block​

• This block enables you to create aggregations for existing columns in your data. It calculates an aggregate value for a property or block using a specified aggregation function. The output is a numerical value.

• When you create a Numeric aggregation block, you need to select a Measure type from the drop-down and the following options are available.
  • Conversion Rate - The percentage of actors that did a conversion event.
  • Average Event Count per Actor - The average number of events performed by each actor.
  • Aggregate over property of an event - Custom aggregate for actors which did at least one of the events.
  • Intervals Engaged - The count of time buckets for each actor, in which that actor met the engaged event criteria.

• When you choose the Conversion Rate and Average Event Count per Actor measures, you need to choose events as the next step.
• When you choose Aggregate over property of an event, you need to select an aggregator and set the value.
• When you choose Intervals Engaged, you need to configure the interval and then select the event.

You can also configure property filters within this block.

Conversion block​

A conversion block segments a dataset based on observed behaviors and associated properties. The behavior of each record in the dataset is determined by the events linked to the entity. The output of the conversion block is the percentage of users who did the events specified. As with the Numeric Aggregation block, you will need to choose a measure type and configure the events for this metric type accordingly.

Ratio block

The Ratio metric block allows you to create ratios between two metric blocks.

With the Ratio Metric block, you can create custom metrics by dividing the total count (total conversions), unique count (unique conversions), total revenue, or total value of one event by the total count (total conversions) or unique count (unique conversions) of another event.

Segmentation Module​

The Segmentation module enables you to choose a cohort of actors, such as users, or one or more attributes to include in the analysis. It has two sub-sections: Performed by and Grouped by that allow users to add cohorts and attributes respectively. You can choose to create a cohort either by choosing an existing cohort from the drop-down or use the + New Cohort option to create a behavioural cohort block in one click.

Filters Module​

Filters can be used to narrow down data in a visualization. Filters make it easier for the user to answer exploratory questions, for example if the user wants to see results for a specific tier, they can define a subscription tier filter and see the narrowed down data.

Analytics also allows users to choose JSON columns in this module. When you click on a JSON column, it expands to display all the keys that are available for that particular column. You can choose a key and click Apply. Once this is done, the end key that is selected will be chosen as the display name for that column.

Visualization Module​

The Visualization module in the Experiment Scorecard template enables you to run and view the analysis in the form of a pivot chart. It also provides the ability to add the chart to a dashboard.

The following features are available in this module:

• Time Range - Allows you to configure time range for the analysis.
• Column Sorting - Allows you to sort the columns in the resulting pivot table.

Time Range section

• At the top of the Visualization window, users will be able to configure the time range and time grain for the analysis. Time Range refers to the complete period of time during which events will be taken into account for the analysis. Examples include the last 2 years or the time range between two specific dates. Time Range is set by default to the duration of the experiment.
• You can set the time range using a simple drop-down or even choose from the quick options and quickly iterate through different choices without leaving the chart. It is also possible to set a lag by clicking Offset and setting the Ending.