Ratio metrics

Ratio metrics are calculated metrics that let you combine data about two different events in a single metric, giving you greater flexibility and letting you create more advanced metrics. Using ratio metrics, you can create custom metrics based on the total count (total conversions), unique count (unique conversions), total revenue, or the total value of one event divided by the total count (total conversions) or unique count (unique conversions) of a second event. 

Calculation methodology

To calculate ratio metrics, Optimizely does the following:

1. Check the denominator event – Optimizely first checks if the user completes the denominator event.
2. Check the numerator events – If the user completes the denominator event, Optimizely then checks for the numerator events within a 48-hour window. The numerator events must occur within 48 hours after the denominator event to be included in the ratio metric calculation.

Examples

Ecommerce website example

For example, you have an ecommerce website and want to track purchases. You are running an A/B test on a product details page. Variation A has a large photo and large text, while variation B has a small photo and small text. You want to track the conversion rate of people who initiate a purchase by adding an item to a cart and then completing the purchase. In this scenario, you have two events to track.

1. Clicking an add-to-cart button
2. Completing the checkout

In this example, the ratio metric is the ratio of completed checkouts (numerator event) divided by the number of add-to-cart clicks (denominator event). 

Example calculation

Using the previous example, this is how the ratio metric, Purchases, is calculated.

• 100 people were bucketed into variation A.
  • From them, 20 people clicked the add-to-cart button.
    • Of them, 17 completed their checkout within 48 hours.
• 150 people were bucketed into variation B.
  • From them, 13 people clicked the add-to-cart button.
    • Of them, 12 people completed their checkout within 48 hours.

Metric 1 calculation (unique add-to-cart click count)

Variation A (large image, large text)

• Unique conversions – 20
• Conversion rate – 20/100 = 20%

Variation B (small image, small text)

• Unique conversions – 13
• Conversion rate – 13/150 = 8%

Metric 2 calculation (ratio metric; the percentage of completed checkouts after adding to the cart)

Variation A (large image, large text)

• Unique conversions for completed_checkout event – 17
• Unique conversions for add_to_cart event – 20
• Conversion rate – 17/20 = 85%

Variation B (small image, small text)

• Unique conversions for completed_checkout event – 12
• Unique conversions for add_to_cart event – 13
• Conversion rate – 12/13 = 92%

Example metric calculation logic

Using the previous ecommerce example, here are four example users, their checkout experiences, and how Optimizely would count their metrics.

• User 1 – Lands on the product details page. Clicks add_to_cart three times. Completes the checkout for these three items. After ten hours, returns to the product details page, clicks add_to_cart one time, and completes their checkout.
• User 2 – Lands on the product details page. Clicks add_to_cart one time and leaves the page. After 14 hours, returns and clicks add_to_cart one time and completes their checkout.
• User 3 – Lands on the product details page. Clicks add_to_cart two times. After three days (72 hours), returns to the product details page, clicks add_to_cart one time, and completes their checkout.
• User 4 – Lands on the product details page. Clicks add_to_cart one time, leaves the page, and never returns or completes their checkout.

Ratio metric calculation (total add_to_cart/total completed_checkout = counting events)

• User 1 – 3 add_to_cart + 1 add_to_cart/1 completed_checkout + 1 completed_checkout = 4 add_to_cart/2 completed_checkout
• User 2 –  1 add_to_cart + 1 add_to_cart = 2 add_to_cart/1 completed_checkout)
• User 3 – 1 add_to_cart/1 completed_checkout. The first two completed_checkout events were over 48 hours from the purchase.
• User 4 – Null. The completed_checkout event did not happen, and completed_checkout is the denominator.

Ratio metric calculation (unique add_to_cart/unique completed_checkout = counting users)

• User 1 – 1 add_to_cart/1 completed_checkout. Even though there were 2 and 4 add_to_cart events, what matters for Optimizely is that there is one user.
• User 2 – 1 add_to_cart/1 completed_checkout
• User 3 – 1 add_to_cart/1 completed_checkout
• User 4 – Null. The completed_checkout event did not happen, and completed_checkout is the denominator.

Statistical methodology

When conducting experiments using ratio metrics, it is essential to estimate the metric's variance to determine its statistical significance. Given that a ratio metric is a ratio of two events, Optimizely employs a first-order Taylor series approximation (often referred to as the Delta method) to approximate this variance. 

For a ratio metric R̂ defined as 

Where

• x_i represents the observed values of the denominator event
• y_i represents the observed values of the numerator event

The approximate variance of R̂ is calculated as

Where

• n is the sample size.
• μ_x is the mean of the denominator variable
• σ_x² is the variance of x.
• σ_y² is the variance of y.
• σ_xy is the covariance between x and y.

This approximation helps in understanding the variability of the ratio metric, which is crucial for hypothesis testing. The presence of covariance (σ_xy) in the formula indicates that the two events in a ratio metric may not be independent. Instead, their values may be statistically dependent, meaning that changes in one event could be correlated with changes in the other. This dependence is captured in the variance calculation to ensure accurate statistical inferences. Optimizely's sequential testing methods were adjusted to account for this variance estimation, ensuring accurate and reliable test results.

Best practices

By following these methodologies and best practices, you can leverage ratio metrics to gain deeper insights into user behavior while maintaining a comprehensive understanding of the underlying data dynamics.

Ratio metrics can serve as your primary, secondary, or monitoring metric. You should create separate secondary metrics for the numerator and the denominator to ensure you understand the value of the ratio metric. This is important because of the following:

• If only one event changes drastically, the ratio metric can become skewed, potentially causing misleading conclusions.
• If the ratio remains the same, but the total counts of both events drop dramatically, it could indicate an underlying issue not apparent from the ratio metric alone.

From the previous ecommerce example, you should create the following:

• A Completed Purchase ratio metric.
• A simple metric based on the completed_checkout event.
• A simple metric based on the add_to_cart event.