Relevant products:
- Optimizely Web Experimentation
- Optimizely Web Personalization
- Optimizely Performance Edge
- Optimizely Feature Experimentation
This topic describes how to:
- Understand how Optimizely Experimentation counts conversion events
- View examples for how conversions are counted under a few specific conditions
Imagine that you are running an experiment on all the product detail pages on your site. A user lands on one of those pages and enters the experiment. They’re bucketed into a certain variation and convert on an event you’re tracking: click the Add-to-Cart button. As the user navigates around your site, they convert a few more times. How exactly does Optimizely Experimentation count those events and display them on your Results page?
This article explains how Optimizely Experimentation counts conversions for a variation or personalized experience; it is Optimizely Experimentation’s attribution model. There are a few important differences in how conversion events are attributed in Experimentation versus Personalization, which this article also explains.
We will also cover how conversion events are counted under a few specific conditions:
-
When you target an experiment to certain users via audience conditions
-
When you filter the results only for a certain audience segment
-
When you filter results only for a certain date range
Jump straight to the general rules. Or jump straight to helpful infographics at the end of this article.
Here is a diagram that follows a single user’s activity on your site:
We will use this example to follow how Optimizely Experimentation’s different products count conversions and events in specific situations.
-
E is a conversion event, such as a click to the Add-to-Cart button.
-
D is a decision point when Optimizely Experimentation decides whether the user is bucketed into a certain variation based on URL and audience targeting conditions. In this example, all decision points in the timeline are for the same variation.
-
A session is a period of activity for a user. An existing session ends, and a new session begins after 30 minutes of inactivity on the site. The maximum session length is 24 hours, at which point a new session automatically begins (even if the user wasn’t inactive for 30 minutes).
-
Unless otherwise noted, we will always assume that the user is shown the same variation in Sessions 1 and 3.
The example in the timeline diagram shows the user’s activity on your site. In the middle of Session 1, the user lands on a URL where the experiment is running, and a decision (D) is made based on targeting conditions. The user is either shown the variation or not and converts two more times. After 30+ minutes of inactivity, a fresh session begins. The user fires the conversion event (E) five times during Session 2. In Session 3, the user converts twice, and a decision (D) is made again -- perhaps that the user has returned to a product detail page, and the decision is re-evaluated. The same user also fires a conversion event (E) five more times during Session 4.
Below, we will use this same timeline to walk through how the conversion events are counted in different products and under certain conditions.
General rules
Here are a few basic rules that Optimizely Experimentation follows when counting conversion events.
|
User scoping versus session scoping |
|
|---|---|
|
User-level scoping When a user is shown a variation, all subsequent conversion events from that user are attributed to that variation for as long as the experiment is active.
In Optimizely Feature Experimentation, there is no "disqualification." If the visitor was previously bucketed into the experiment and later they are evaluated again but then they do not pass the Audience Conditions (not logged in anymore, for example), then no decision event is sent by the SDK, so Optimizely Feature Experimentation will never know that the visitor no longer qualifies or sees the experiment. This user's conversions continue to be attributed to the experiment. |
Products with user-level scoping:
|
|
Session-level scoping When a user is shown a variation, all subsequent conversion events fired during that user session are attributed to that variation. |
Products with session-level scoping:
|
For Optimizely Feature Experimentation, in situations where a user encounters multiple variations for the same experiment, all subsequent conversions are attributed to the first variation encountered within the date range of the results query.
This means that a user's conversions can be attributed to different variations depending on the date range of your results query.
For example, a user encountered Variation A at T1, then encountered Variation B at T2, and finally converted at T3. If your results query spans from T1 to T3, the conversion will be attributed to Variation A. If your results query spans from T2 to T3, the conversion will be attributed to Variation B.
|
Common counting scenarios |
|
|
Example: A user is bucketed into an experiment that targets signed-in visitors and is shown a variation. The same user returns the following day, triggering another decision for that experiment. She no longer qualifies because she is not signed in. Optimizely Experimentation attributes all events from this user from the moment she was bucketed until she's disqualified from the experiment. |
|
|
If the user is in a given segment at the end of a session, then all events in that session are attributed to that segment when results are filtered by that segment. |
|
|
If the session starts in a given date range, then all events are counted when results are filtered by that date range. |
|
|
Events in new sessions that begin after the reset point are counted. |
|
Below are a few examples that explain how events are scoped to the user or session level and how this impacts counting:
Optimizely Web Experimentation, except Optimizely Web Personalization
Use user-scoped results. The timeline below shows how conversion events (E) for a single user are counted when results are scoped to the user.
Optimizely Web Experimentation counts 16 total conversion events (E) and attributes them to one user. This is user-scoped event attribution. When the user is shown a variation, all subsequent conversion events from that user are attributed to that variation for as long as the experiment is active.
|
Users = 1 |
Total conversions = 16 |
Unique converting users = 1 |
Optimizely Web Personalization
Optimizely Web personalization is the only product that has session-scoped results. Here is the same event timeline as above, but counted by session instead of a user.
When the user is shown a variation, all subsequent conversion events that are fired during that user session are attributed to that variation. Optimizely Web Personalization uses session-scoped metrics in Personalization because it is not uncommon for a single user to see several different personalized experiences over time in behaviorally-targeted campaigns.
|
Sessions = 2 |
Total conversions = 4 |
Unique converting sessions = 2 |
Next, see how event counting works under several specific conditions: including or excluding users based on audience conditions, filtering the Results page by segment, and filtering the Results page by time range.
Audience targeting
Optimizely Experimentation lets you target experiments and personalized experiences to groups of users (called audiences) who share attributes you define. This section explains how Optimizely Experimentation attributes a single user’s events over time as they navigate the site.
When the user first visits the site, Optimizely Experimentation buckets them as a signed-in visitor and shows them a variation during Session 1. When the user returns the following day, they trigger another decision for that experiment in Session 3 and no longer qualify because they are now signed out.
Example for all Optimizely Experimentation products, except Optimizely Web Personalization:
All user-scoped Optimizely Experimentation products attribute all events from this user from the moment they are bucketed (Session 1) until the moment they are disqualified from the experiment (Session 3).
|
Users = 1 |
Total conversions = 9 |
Unique converting users = 1 |
Example for Optimizely Web Personalization:
During Session 1, the user is bucketed an experiment targeting signed-in visitors. Only the subsequent events fired during that user session are attributed to the variation.
|
Sessions = 1 |
Total conversions = 2 |
Unique converting sessions = 1 |
"Purchase" event example
Imagine again that a visitor to your site is bucketed into a variation during Session 1. The visitor triggers a decision for the experiment in Session 3, which means they no longer qualify. Now, suppose this visitor makes a purchase after they're disqualified in Session 3.
Events are no longer attributed to this visitor after they're disqualified, so the unique converting user count would be 0. Optimizely Web Personalization doesn't attribute events to visitors after they're disqualified from an experiment.
Filtering by segment
When you look at your results in Optimizely Experimentation, you can use segmentation to see how a subset of users respond to the variation (instead of your users overall). This section explains how Optimizely Experimentation attributes conversion events when filtering your results by segment. If the user is in a given segment at the end of a session, then all events in that session are attributed to that segment when results are filtered by that segment.
Example for all Optimizely Experimentation products, except Optimizely Web Personalization:
|
Users = 1 |
Total conversions = 11 |
Unique converting users = 1 |
Example for Optimizely Web Personalization:
When the user is in a given segment at the end of a session (like Sessions 1 and 3), all subsequent conversion events that are fired during that specific user session are attributed to the variation.
|
Sessions = 2 |
Total conversions = 4 |
Unique converting sessions = 2 |
Filtering by date range
In Optimizely Experimentation, you can also filter your results by date range. For example, you might want to see conversions for Jan. 1 - Jan. 20 on your Results page. This section explains how Optimizely Experimentation attributes conversion events when filtering your results by date range.
The basic counting rule for date ranges is that if a session starts in the filter’s date range, then all events in that session are counted when results are filtered by that date range.
Example for Optimizely Web Experimentation, except Optimizely Web Personalization:
If the session starts in the given date range (like Sessions 2, 3, and 4), all events in that session are counted when results are filtered by that time range.
|
Users = 1 |
Total conversions = 14 |
Unique converting users = 1 |
Example for Optimizely Web Personalization:
When a user is bucketed into the experiment, and the session starts in the date range (Session 3), all subsequent conversion events that are fired during that user session are attributed to the variation.
|
Sessions = 1 |
Total conversions = 2 |
Unique converting sessions = 1 |
Example in Optimizely Feature Experimentation:
Like all Optimizely Experimentation products except Optimizely Web Personalization, Optimizely Feature Experimentation counts events based on user-level scoping. If the session starts in the date range (like Sessions 3 and 4), all subsequent conversion events that are fired after the user is bucketed into the experiment are attributed to the variation
Resetting results
In Optimizely Experimentation, you can reset your Results page. This section explains how a user’s events are attributed to an experiment or campaign when you reset the page.
When you reset your results, all sessions that began before the reset point are discarded, even if those sessions persist beyond that point.
Here is how to reset the results page for Optimizely Web Experimentation and Optimizely Web Personalization.
The basic rule is that events in new sessions that begin after the reset point are counted.
Example for Optimizely Web Experimentation:
|
Users = 1 |
Total conversions = 9 |
Unique converting users = 1 |
Event tracking and network requests
When monitoring your network traffic, you will see event-tracking calls for visitors to your page. When a visitor triggers an event in Optimizely Experimentation, the event fires a tracking call, which is picked up in network traffic.
You may see that Optimizely Experimentation tracks events for visitors who are not currently bucketed into an experiment and tracks events that are not currently part of an experiment. Wondering why? This means you can retroactively add metrics to your experiments and still get data starting the day an event was created. It also lets you target visitors with behavioral targeting and measure the reach of your campaigns.
To ensure privacy for your visitors, you can anonymize visitor IP addresses. Learn more about privacy settings in Optimizely Web Experimentation.
Helpful Infographics
Here is an overview of how Optimizely Experimentation counts Total Conversions:
Here is an overview of how Optimizely Experimentation counts Unique Conversions:
<%= heading %>