- Optimizely Web Experimentation
- Optimizely Web Personalization
- Optimizely Performance Edge
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
Imagine that you are running an experiment on the product detail pages on your site. A user lands on one of those pages, enters the experiment, is bucketed into a certain variation, and converts on an event you are tracking: click Add to Cart. The user converts a few times navigating your site.
Optimizely Experimentation's attribution model counts conversions for a variation or personalized experience. There are a few important differences in how conversion events are attributed in Experimentation versus Personalization, which this article also explains.
This article also explains how conversion events are counted under a few specific conditions:
-
When you target an experiment to certain users via audience conditions.
-
When you filter the results only for a certain audience segment.
-
When you filter results only for a certain date range.
The following diagram follows a single user's activity on your site:
-
E is a conversion event, such as clicking Add to Cart.
-
D is a decision point when Optimizely Experimentation decides whether the user is bucketed into a certain variation based on URL and audience targeting conditions. In this example, decision points in the timeline are for the same variation.
-
A session is a period of activity for a user. An existing session ends, and a new session begins after 30 minutes of inactivity on the site for Web Experimentation. The maximum session length is 24 hours, at which point a new session automatically begins (even if the user was not inactive for 30 minutes).
-
Unless otherwise noted, assume that the user is shown the same variation in Sessions 1 and 3.
The example in the timeline diagram shows the user's activity on your site:
- In the middle of Session 1, the user lands on a URL where the experiment is running, and a decision (D) is made based on targeting conditions. The user is either shown the variation or not and converts two more times. After 30+ minutes of inactivity, a fresh session begins.
- The user fires the conversion event (E) five times during Session 2.
- In Session 3, the user converts twice, and a decision (D) is made again–perhaps that the user has returned to a product detail page, and the decision is re-evaluated.
- The same user also fires a conversion event (E) five more times during Session 4.
The timeline in this example is used to describe how the conversion events are counted in different products and under certain conditions.
User scoping versus session scoping
User-level scoping
- Optimizely Web Experimentation
- Optimizely Feature Experimentation
When a user is shown a variation, subsequent conversion events from that user are attributed to that variation for as long as the experiment is active.
- Optimizely Web Experimentation – user can be disqualified from the variation, and conversions are no longer attributed after disqualification.
- Optimizely Feature Experimentation – user cannot get disqualified from the variation, and conversions continue to be attributed.
In Optimizely Feature Experimentation, there is no "disqualification." If the user was previously bucketed into the experiment and is later evaluated again but does not pass the Audience Conditions (not logged in anymore, for example), then no decision event is sent by the SDK. Therefore, Optimizely Feature Experimentation cannot know that the visitor no longer qualifies or sees the experiment. This user's conversions continue to be attributed to the experiment.
For Optimizely Feature Experimentation, in situations where a user encounters multiple variations for the same experiment, subsequent conversions are attributed to the first variation encountered within the date range of the results query. This means that a user's conversions can be attributed to different variations, depending on the date range of your results query. For example, a user encountered Variation A at T1, then encountered Variation B at T2, and finally converted at T3. If your results query spans from T1 to T3, the conversion is attributed to Variation A. If your results query spans from T2 to T3, the conversion is attributed to Variation B.
Session-level scoping
- Optimizely Web Personalization
When a user is shown a variation, subsequent conversion events fired during that user session are attributed to that variation.
Common counting scenarios
- Audience targeting – Example: A user is bucketed into an experiment that targets signed-in visitors and is shown a variation. The same user returns the following day, triggering another decision for that experiment. She no longer qualifies because she is not signed in. Optimizely Experimentation attributes events from this user from the moment she was bucketed until she is disqualified from the experiment.
- Filtering by segment – If the user is in a given segment at the end of a session, then events in that session are attributed to that segment when results are filtered by that segment.
- Filtering by date range – If the session starts in a given date range, then events are counted when results are filtered by that date range.
- Resetting results – Events in new sessions that begin after the reset point are counted.
The following sections explain how events are scoped to the user or session level and how this impacts counting.
Optimizely Web Experimentation (except Optimizely Web Personalization)
Use user-scoped results. The following timeline shows how conversion events (E) for a single user are counted when results are scoped to the user.
Optimizely Web Experimentation counts 16 total conversion events (E) and attributes them to one user. This is user-scoped event attribution. When the user is shown a variation, subsequent conversion events from that user are attributed to that variation for as long as the experiment is active.
- Users – 1
- Total conversions – 16
- Unique converting users – 1
Optimizely Web Personalization
Optimizely Web personalization is the only product that has session-scoped results. Here is the same event timeline as above, but counted by session instead of a user.
When the user is shown a variation, subsequent conversion events that are fired during that user session are attributed to that variation. Optimizely Web Personalization uses session-scoped metrics in Personalization because a single user may often see several different personalized experiences over time in behaviorally-targeted campaigns.
- Sessions – 2
- Total conversions – 4
- Unique converting sessions – 2
The following sections describe how event counting works under several specific conditions: including or excluding users based on audience conditions, filtering the Results page by segment, and filtering the Results page by time range.
Audience targeting
Optimizely Experimentation lets you target experiments and personalized experiences to groups of users (called audiences) who share attributes you define. This section explains how Optimizely Experimentation attributes a single user's events over time as they navigate the site.
When the user first visits the site, Optimizely Experimentation buckets them as a signed-in visitor and shows them a variation during Session 1. When the user returns the following day, they trigger another decision for that experiment in Session 3 and no longer qualify because they are now signed out.
Example for Optimizely Experimentation products (except Optimizely Web Personalization)
User-scoped Optimizely Experimentation products attribute events from this user from the moment they are bucketed (Session 1) until the moment they are disqualified from the experiment (Session 3).
- Users – 1
- Total conversions – 9
- Unique converting users – 1
Example for Optimizely Web Personalization
During Session 1, the user is bucketed an experiment targeting signed-in visitors. Only the subsequent events fired during that user session are attributed to the variation.
- Sessions – 1
- Total conversions – 2
- Unique converting sessions – 1
Purchase event example
Imagine again that a visitor to your site is bucketed into a variation during Session 1. The visitor triggers a decision for the experiment in Session 3, which means they no longer qualify. Now, suppose this visitor makes a purchase after they are disqualified in Session 3.
Events are no longer attributed to this visitor after they are disqualified, so the unique converting user count would be 0. Optimizely Web Personalization does not attribute events to visitors after they are disqualified from an experiment.
Filter by segment
When you look at your results in Optimizely Experimentation, you can use segmentation in Optimizely Web Experimentation or segmentation in Optimizely Feature Experimentation to see how a subset of users respond to the variation (instead of your users overall). This section explains how Optimizely Experimentation attributes conversion events when filtering your results by segment. If the user is in a given segment at the end of a session, then events in that session are attributed to that segment when results are filtered by that segment.
Example for Optimizely Experimentation products (except Optimizely Web Personalization)
- Users – 1
- Total conversions – 11
- Unique converting users – 1
Example for Optimizely Web Personalization
When the user is in a given segment at the end of a session (like Sessions 1 and 3), subsequent conversion events that are fired during that specific user session are attributed to the variation.
- Sessions – 2
- Total conversions – 4
- Unique converting sessions – 2
Filter by date range
In Optimizely Experimentation, you can also filter your results by date range. For example, you might want to see conversions for Jan. 1 - Jan. 20 on your Results page. This section explains how Optimizely Experimentation attributes conversion events when filtering your results by date range.
The basic counting rule for date ranges is that if a session starts in the filter's date range, then events in that session are counted when results are filtered by that date range.
Example for Optimizely Web Experimentation, except Optimizely Web Personalization
If the session starts in the given date range (like Sessions 2, 3, and 4), events in that session are counted when results are filtered by that time range.
- Users – 1
- Total conversions – 14
- Unique converting users – 1
Example for Optimizely Web Personalization
When a user is bucketed into the experiment, and the session starts in the date range (Session 3), subsequent conversion events that are fired during that user session are attributed to the variation.
- Sessions – 1
- Total conversions – 2
- Unique converting sessions – 1
Example in Optimizely Feature Experimentation
Like all Optimizely Experimentation products except Optimizely Web Personalization, Optimizely Feature Experimentation counts events based on user-level scoping. If the session starts in the date range (like Sessions 3 and 4), subsequent conversion events that are fired after the user is bucketed into the experiment are attributed to the variation
Reset results
In Optimizely Experimentation, you can reset your results page. This section explains how a user's events are attributed to an experiment or campaign when you reset the page.
When you reset your results, sessions that began before the reset point are discarded, even if those sessions persist beyond that point for Optimizely Web Experimentation and Optimizely Web Personalization.
The basic rule is that events in new sessions that begin after the reset point are counted.
Example for Optimizely Web Experimentation
- Users – 1
- Total conversions – 9
- Unique converting users – 1
Event tracking and network requests
When monitoring your network traffic, you will see event-tracking calls for visitors to your page. When a visitor triggers an event in Optimizely Experimentation, the event fires a tracking call, which is picked up in network traffic.
You may see that Optimizely Experimentation tracks events for visitors who are not currently bucketed into an experiment and tracks events that are not currently part of an experiment. Wondering why? This means you can retroactively add metrics to your experiments and still get data starting the day an event was created. It also lets you target visitors with behavioral targeting and measure the reach of your campaigns.
To ensure privacy for your visitors, you can anonymize visitor IP addresses. Learn more about privacy settings in Optimizely Web Experimentation.
The following image shows how Optimizely Experimentation counts Total Conversions:
The following image shows how Optimizely Experimentation counts Unique Conversions:
<%= heading %>
