The Ratio metric lets you create custom metrics by calculating the ratio of two different events. For example, in an online streaming app, you can define a ratio metric to measure the completion rate by calculating the proportion of content that users watched till the end compared to the total content they started. This metric helps assess viewer engagement and identify trends in content consumption.
Example use case
Consider a streaming platform evaluating the ratio of total user watch time to total revenue. In this scenario, you configure the following numerator and denominator event values:
- Numerator Event – Select Total Time Viewed (TTV).
- Denominator Event – Select Total Revenue.
By establishing a ratio metric of Total Time Viewed to Total Revenue, you can assess how effectively user engagement translates into revenue. A higher ratio suggests strong viewer retention relative to earnings, while a lower ratio may indicate opportunities to optimize monetization strategies.
Configuration
Within this metric block, you must define a numerator and a denominator. The numerator and the denominator are calculated separately before the ratio is derived.
Set the numerator
For the numerator, choose one of the following options:
- Create a conversion metric
- Create a numeric aggregation metric
- Select an existing metric
Set the denominator
For the denominator, choose one of the following options:
- Create a conversion metric
- Create a numeric aggregation metric
- Select an existing metric
Statistical methodology
When conducting experiments using ratio metrics, it is essential to estimate the metric's variance to determine its statistical significance. Given that a ratio metric is a ratio of two events, Optimizely employs a first-order Taylor series approximation (often referred to as the Delta method) to approximate this variance.
For a ratio metric R̂ defined as
Where
- xi represents the observed values of the denominator event
- yi represents the observed values of the numerator event
The approximate variance of R̂ is calculated as
Where
- n is the sample size.
- μx is the mean of the denominator variable
- σx2 is the variance of x.
- σy2 is the variance of y.
- σxy is the covariance between x and y.
This approximation helps in understanding the variability of the ratio metric, which is crucial for hypothesis testing. The presence of covariance (σxy) in the formula indicates that the two events in a ratio metric may not be independent. Instead, their values may be statistically dependent, meaning that changes in one event could be correlated with changes in the other. This dependence is captured in the variance calculation to ensure accurate statistical inferences. Optimizely's sequential testing methods were adjusted to account for this variance estimation, ensuring accurate and reliable test results.
Please sign in to leave a comment.