Primary metrics, secondary metrics, and monitoring goals

The right metrics can validate or disprove your hypothesis and help you progress toward your business goals. In experience optimization, metrics have different roles depending on where you set them and what you want to know.

• Primary metric – Determines whether the test wins or loses. It tracks how your changes affect visitor behavior.
• Secondary metrics – Provide additional information about visitor behavior related to your change and across your site.
• Monitoring goals – Additional guardrail metrics that are not your primary or secondary metrics. They help you gather insights that are key to long-term success.

The following are a few tips for setting goals and metrics:

• Focus on a direct visitor action that is on the same page as the changes you make.

• Consider how your changes affect other parts of your site. Create metrics to measure potential interaction to see if your test moves customers in the right direction.

• Place different metrics at different points in your funnel to gather timely data about visitor behavior.

Primary metric

In Optimizely, you set a primary metric for each experiment or campaign to determine its success. It is the most important goal of the experiment and proves or disproves your hypothesis.

Use the following questions to choose a primary metric:

• What visitor action indicates that this variation is a success? – Measure the action visitors take as a result of this test.
• Does this event directly measure the behavior you are trying to influence? – Use an action directly affected by your change to decide whether your test helped.
• Does the event fully capture the behavior you are trying to influence? – Consider whether your primary metric captures the behavior you are trying to influence.

Consider setting a primary metric on the same page as your change. Many optimization teams automatically track revenue per visitor as the primary metric. Top-level metrics like revenue and conversion rate are important, but these events are often far away from the changes made, and your test may take a long time to reach statistical significance or end up inconclusive. 

Stats Engine uses false discovery rate control to help you make better business decisions. The primary metric's false discovery rate control is done in a way that is independent of secondary and monitoring metrics. This means that the primary metric's evaluation is separate and independent of other metrics. See the Stats Engine approach to metrics and goals section for information.

Example 1

For example, you test the design and placement of an Add-to-Cart button. Your business cares about revenue, which is measured five pages down the funnel. You may devote a lot of traffic to this test and risk an inconclusive result.

Instead, you decide to measure clicks to the Add-to-Cart on product pages. This is a primary metric directly affected by the changes you make. 

Example 2

If you are testing bulk discounts on your site, your primary metric might be conversion rate or average order value (AOV). Neither metric fully accounts for the behavior you are trying to affect.

The conversion rate could rise as customers are incentivized or decrease as customers wait to create large, discounted orders. AOV could rise as customers buy more in bulk or decrease as discounts replace full-price orders.

From this perspective, revenue-per-visitor is the best metric. It equals the conversion rate (how often customers purchase) multiplied by the AOV (how much they spend). It is the best overarching goal in this test, where smaller goals may provide conflicting information.

Secondary metrics

Secondary metrics help you gather insights for long-term success. Optimizely lets you rank secondary metrics from two to five. Secondary metrics track long-distance events and more ambitious metrics, such as end-of-funnel events like order value and order confirmation. These events provide valuable information but may reach significance slower as a primary metric.

Secondary metrics also give you visibility across the different funnel steps. The significance threshold for secondary metrics is adjusted based on the number of metrics and variations to ensure proper false discovery rate control. This has the consequence that the more secondary metrics you have, the longer it may take for each to reach statistical significance. Secondary metrics do not impact the primary metric's speed to significance because the primary metric is evaluated separately. See the Stats Engine approach to metrics and goals section for information.

Example

For example, if you change your product page and display shipping costs, your secondary metric might measure the change in drop-offs from the shipping page in your funnel. Use secondary metrics to learn when visitors drop off or return to the home page and how these patterns compare between the original and variations.

Here is a list of common secondary metrics and the reason for tracking:

• Searches submitted – See how many searches are submitted.
• Category pageview – Discover whether visitors go to the site through category pages.
• Subcategory pageview – Learn whether visitors reach subcategory pages.
• Product pageview – Know the percentage of visitors who do or do not view a product during a visit.
• Add-to-cart – Understand what percentage of visitors add-to-cart per test, category, or product type. 
• Shopping cart pageview – See how many visitors progress to the shopping cart.
• Checkout pageview – Understand how many visitors continue from the shopping cart to checkout. 
• Payment pageview – Learn what percentage of visitors continue from checkout to payment.
• Conversion rate – Know what percentage of visitors ultimately convert or complete payment.

Monitoring goals

Monitoring goals are goals and events that are not your primary or secondary metrics. They help you gather insights that are key to long-term success but are diagnostic. Monitoring goals track whether your experiment moves visitors in the right direction. Your change might create adverse effects in another metric. Monitoring goals form a warning system that alerts you when you hurt another revenue path.

The false discovery rate control calculations give each monitoring metric a fractional weight. Monitoring metrics are evaluated with a weight of 1/n, where n is the number of monitoring metrics. For example, if there are two monitoring metrics, each would be given a weight of 1/2. This approach ensures that monitoring metrics have minimal impact on the evaluation of secondary metrics and no impact on the primary metric. See the Stats Engine approach to metrics and goals section for information.

Example

For example, you show visitors more products on the product category page. With your primary metric, you find that people view more products. Here are some other questions the monitoring goal can help you find out:

• Are people more price-conservative when presented with more products? (Average order value)
• Are people buying more products? (Conversion rate)
• Are people frustrated and unable to find what they are looking for? (Subcategory filters)

Here are common monitoring goals and their reason for tracking:

• Search bar opened – Learn what percentage of search bar interactions do not lead to submissions.
• Top menu Clickthrough rate (CTR) – Discover how often visitors use the top menu per page or step in the funnel.
• Home page CTR – See how often visitors exit to the Home page from any given page. 
• Category page filter usage – Understand the frequency of filter usage.
• Product page quantity selection – Understand the percentage of visitors interacting with quantity selection.
• Product page more info – Understand how many visitors seek information about a product
• Product page tabs – Discover how often visitors interact with each tab.
• Payment type chosen – See which payment type users prefer per experiment.
• Return or back button CTR – Learn how often visitors exit a page with a specific button.

Stats Engine approach to metrics and goals

When you run an experiment with many variations and metrics using traditional statistics, you increase the chance of false positive results. Your test data may show a significant difference between your original and your variation, but it is just noise in the data. This makes it harder to declare winners.

Stats Engine uses false discovery rate control to reduce your chance of making an incorrect business decision or implementing a false positive among conclusive results.

• The first ranked metric is your primary metric – The primary metric's false discovery rate control is done in a way that is independent of secondary and monitoring metrics. This means that the primary metric's evaluation is separate and independent of other metrics.
• Metrics ranked two through five are considered secondary metrics – The significance threshold for secondary metrics is adjusted based on the number of metrics and variations to ensure proper false discovery rate control. This has the consequence that the more secondary metrics you have, the longer it may take for each to reach statistical significance. Secondary metrics do not impact the primary metric's speed to significance because the primary metric is evaluated separately. 
• Any metrics ranked beyond the first five are monitoring metrics – The false discovery rate control calculations give each monitoring metric a fractional weight. Monitoring metrics are evaluated with a weight of 1/n, where n is the number of monitoring metrics. For example, if there are two monitoring metrics, each would be given a weight of 1/2. This approach ensures that monitoring metrics have minimal impact on the evaluation of secondary metrics and no impact on the primary metric.

Revenue goals and skew correction with Stats Engine

Stats Engine works the same for revenue-per-visitor goals as for other goals. You can look at your results and get an accurate assessment of your error rates on winners and losers and confidence intervals on the average revenue per visitor (RPV).

However, when interpreting your results for RPV goals, you should be aware of some differences. Testing for a difference in average revenue between a variation and baseline is more challenging because revenue distributions are often heavily skewed. This impedes the distributional results for many techniques, including t-tests and Stats Engine. They have less power to detect differences in average revenue when those differences exist.

Optimizely's Stats Engine regains some of this lost power through skew correction. Skew corrections were designed to work well with other aspects of Stats Engine. The underlying skewness of the distributions is correctly factored into the shape of the confidence interval. Detecting differences in average revenue is more reasonable for the visitor counts that Optimizely regularly sees in A/B tests.

Consider speed and impact

In a funnel, the most immediate effects are directly downstream from your changes. The measurable impact is bigger when an event is closer to the change. As you move downstream, the signal lessens as visitors from different paths and motivations enter the stream. The effect may not be measurable at the end of the funnel.

Metrics with lower conversion rates require more visitors to reach statistical significance. Events further from the page have a lower improvement in conversion rates as visitors enter from different paths, leave the site before they convert, and more. Your test will take longer to reach significance.

Consider setting a primary metric on the same page as your change. This finds the impact of your change immediately, so you can find a winning variation faster. Quick wins help generate credibility and interest in your testing program and provide fast, reliable insights about visitor behavior. You can build a data-rich testing program to iterate on its insights.

Revenue and conversion rate make excellent secondary metrics and help keep your program focused on long-term success.