Ratio metric

  • Updated

The ratio metric in Warehouse-Native Experimentation Analytics lets you create custom metrics by calculating the ratio of two events. For example, in an online streaming app, you can define a ratio metric to measure the completion rate by calculating the proportion of content users watched until the end compared to the total content they started. This metric helps assess viewer engagement and identify trends in content consumption.

Example use case

Consider a streaming platform evaluating the total user watch time ratio to total revenue. You configure the following numerator and denominator event values:

  • Numerator Event – Select Total Time Viewed (TTV).
  • Denominator Event – Select Total Revenue.

You can assess how effectively user engagement translates into revenue by establishing a ratio metric of Total Time Viewed to Total Revenue. A higher ratio suggests strong viewer retention relative to earnings, while a lower ratio may indicate opportunities to optimize monetization strategies.

Configuration

Within this metric block, you must define a numerator and a denominator. The numerator and the denominator are calculated separately before the ratio is derived. 

Set the numerator

For the numerator, choose one of the following options:

rationumerator.png

Set the denominator

For the denominator, choose one of the following options:

ratiodenomniator.png

Statistical methodology 

When conducting experiments using ratio metrics, you must estimate the metric's variance to determine its statistical significance. Given that a ratio metric is a ratio of two events, Optimizely employs a first-order Taylor series approximation (often called the Delta method) to approximate this variance. 

For a ratio metric R̂ defined as 

Where

  • xi represents the observed values of the denominator event
  • yi represents the observed values of the numerator event

The approximate variance of R̂ is calculated as

Where

  • n is the sample size.
  • μx is the mean of the denominator variable
  • σx2 is the variance of x.
  • σy2 is the variance of y.
  • σxy is the covariance between x and y.

This approximation helps you understand the variability of the ratio metric, which is crucial for hypothesis testing. The covariance (σxy) in the formula indicates that the two events in a ratio metric may not be independent. Instead, their values may be statistically dependent, meaning that changes in one event could be correlated with changes in the other. This dependence is captured in the variance calculation to ensure accurate statistical inferences. Optimizely's sequential testing methods were adjusted to account for this variance estimation, ensuring accurate and reliable test results.