Contextual Bandit Results page

The Contextual Bandit Results page lets you track your contextual bandits' performance, view different experiences and audiences to examine visitor behavior, and take action. Contextual bandits score variations based on the impact on the primary metric and user attributes to serve the most personalized variation to every user.

Contextual bandits use decision-level scoping for event attribution. This means Optimizely assigns events to the decision they happened in. In other words, Optimizely associates events with the decision event that occurred before the event. Optimizely uses decision-level scoping because while the contextual bandit is running, user context can change, and the contextual bandit would optimize for this user, using their new context. Optimizely uses decision-level scoping to capture these details. See the Decision-level scope section in the How Optimizely Experimentation counts conversions documentation for information.

Optimizely calculates most metrics using a session scope within a Personalization campaign. For contextual bandits, Optimizely automatically changes this into decision scope, as it is the best way to capture their performance. Other metrics (for example, revenue per conversion and value per conversion) are still calculated per conversion, even though the UI says decisions. You should not use these metrics for contextual bandits.

View results for contextual bandits

You can view your contextual bandit results from the Campaign Results page.

To access the Campaign Results page, complete one of the following:

• From the Optimizations dashboard, click Results.

  

• From the Manage Campaign dashboard, click Results.

  

Then, in the Experiences section, click on a contextual bandit to open its results page.

About the Contextual bandits results page

The contextual bandit results page includes the following sections:

1. Segmentation drop-down lists – Select different options to see how different segments of users behave on different dates.
2. Download, Share, Edit – Download results as a PDF, copy a shareable URL, or edit the experiences.
3. Summary – Displays the total number of visitors and decisions for the primary metric. A higher number of decisions than visitors suggests the contextual bandit is helpful.
4. Distribution – Shows how the contextual bandit split traffic between users who saw randomized variations and those who received personalized ones.
5. Top User Attributes – Lists user attributes that the contextual bandit model considered the most important when assigning users to variations. Put another way, the "weight" of the attribute when deciding to put a user into a specific variation. These attributes influenced the model's decisions the most.
6. Download Results – Download the full list of user attributes provided to the contextual bandit model with their weights (relative importance).
7. Metrics – Displays how each variation performed for the selected metric, including Unique Conversions, Number of Decisions, and Conversion Rate.
8. View or Hide Graph – Expand or collapse the corresponding metric's graph.
9. Graph type selector – Use the drop-down list to switch between available graphs, including Decisions Over Time, Unique Conversions Over Time, and Unique Conversions per Decision Over Time.
10. Graph details – Hover over the graph to inspect how each variation performs.
11. Additional metrics – View performance for the remaining metrics in your contextual bandit.

Attribute values per variation

Click View Contributing Attributes in the metrics section to see the attributes and values associated with that variation. The expanded view shows a sub-table with two columns:

• Contributing Attribute – The attribute name (for example, Device).

• Attribute Value – The value within that attribute that was associated with conversions for the variation (for example, desktop).

  

For example, a variation named Bronze Credit Card might show the following contributing attributes and attribute values:

• age_group – 21–30
• annual_income_group – $30,000–$45,000
• own_credit_card – Yes

You can read this table as the segment that responded best to the Bronze Credit Card variation. Visitors aged 21 to 30, with an annual income between $30,000 and $45,000, who own a credit card converted at a higher rate on this variation than on others.

Each variation shows its own attributes and values, so two variations in the same experience can list different results.

Metrics reflect the primary metric

The summary, the metrics columns, and the over-time graph all reflect the primary metric configured on the experience. The metric names on the page change with the primary metric you select.

When the primary metric is Overall Revenue, the page shows the following:

• The summary shows Overall Revenue, Total Sessions, and Improvement.
• Metrics section shows revenue, number of decisions, revenue per session, and improvement for each variation.
• The over-time graph is named Revenue Over Time and plots cumulative revenue for each variation against the holdback group.

When the primary metric is a conversion event, the same areas show Unique Conversions, Number of Decisions, and Conversion Rate, and the over-time graphs include Decisions Over Time, Unique Conversions Over Time, and Unique Conversions per Decision Over Time.

The over-time graph plots the cumulative value of the primary metric across the campaign duration, with one line per variation and one line for the holdback group. The holdback group is the share of visitors excluded from the personalized experience, used to measure improvement. Hover any point on a line to see the cumulative value for that variation on that date.