Ratio metrics

Ratio metrics are calculated metrics that let you combine data about two different events in a single metric, giving you greater flexibility and letting you create more advanced metrics. Using ratio metrics, you can create custom metrics based on the total count (total conversions), unique count (unique conversions), total revenue, or the total value of one event divided by the total count (total conversions) or unique count (unique conversions), total revenue, or the total value of a second event. 

Calculation methodology

To calculate ratio metrics, Optimizely does the following:

1. Check the denominator event – Optimizely first checks if the user completes the denominator event.
2. Check the numerator events – If the user completes the denominator event, Optimizely then checks for the numerator events within a 48-hour window. The numerator events must occur within 48 hours after the denominator event to be included in the ratio metric calculation.

When using revenue or another value-based metric as the numerator, Optimizely Experimentation applies sum aggregation. Optimizely sums the numeric values in the numerator across all relevant events before calculating the ratio.

Example

For example, you have an ecommerce website and want to track purchases. You are running an A/B test on a product details page. Variation A has a large photo and large text, while variation B has a small photo and small text. You want to track revenue per add-to-cart click. In this scenario, you have the following two events to track:

1. Purchase completion (total revenue).
2. Add to cart clicked (total conversions).

In this example, the ratio metric is calculated as total revenue from completed purchases / total add-to-cart clicks. 

Example calculation

Optimizely computes this metric by completing the following:

• Checking if the user clicked add to cart (denominator event).
• Summing the revenue from completed purchases (numerator event) within 48 hours.
• Calculating the ratio per user and then averaging across all users.

Use ratio metrics

While the primary use case for ratio metrics is based on sum / count, Optimizely Experimentation also supports different configurations, such as count / count. For example, you could create a ratio metric to track the percentage of completed purchases after an add-to-cart event, using the following structure:

Total completed purchases / Total add-to-cart clicks.

Statistical methodology

When conducting experiments using ratio metrics, it is essential to estimate the metric's variance to determine its statistical significance. Given that a ratio metric is a ratio of two events, Optimizely employs a first-order Taylor series approximation (often referred to as the Delta method) to approximate this variance. 

For a ratio metric R̂ defined as 

Where

• x_i represents the observed values of the denominator event
• y_i represents the observed values of the numerator event

The approximate variance of R̂ is calculated as

Where

• n is the sample size.
• μ_x is the mean of the denominator variable
• σ_x² is the variance of x.
• σ_y² is the variance of y.
• σ_xy is the covariance between x and y.

This approximation helps in understanding the variability of the ratio metric, which is crucial for hypothesis testing. The presence of covariance (σ_xy) in the formula indicates that the two events in a ratio metric may not be independent. Instead, their values may be statistically dependent, meaning that changes in one event could be correlated with changes in the other. This dependence is captured in the variance calculation to ensure accurate statistical inferences. Optimizely's sequential testing methods were adjusted to account for this variance estimation, ensuring accurate and reliable test results.

Best practices

By following these methodologies and best practices, you can leverage ratio metrics to gain deeper insights into user behavior while maintaining a comprehensive understanding of the underlying data dynamics.

Ratio metrics can serve as your primary, secondary, or monitoring metric. You should create separate secondary metrics for the numerator and the denominator to ensure you understand the value of the ratio metric. This is important because of the following:

• If only one event changes drastically, the ratio metric can become skewed, potentially causing misleading conclusions.
• If the ratio remains the same, but the total counts of both events drop dramatically, it could indicate an underlying issue not apparent from the ratio metric alone.

From the previous ecommerce example, you should create the following:

• A Revenue per add-to-cart click (sum revenue / count add-to-cart clicks) ratio metric.
• A standalone metric based on the purchase-completed (total revenue) event.
• A standalone metric based on the total add-to-cart-clicked event.