- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
- Optimizely Web Experimentation
- Optimizely Performance Edge
This article describes how impressions work at Optimizely Experimentation, similar to that used by online media, but the nature of our technology introduces some key differences.
Impression in Optimizely Web Experimentation
In Optimizely Web Experimentation, impression is counted for every Optimizely Web Experimentation page, each time a visitor sees an Optimizely Web Experimentation experience as the result of a test or personalization campaign.
Impressions are the unit of measurement for usage in Optimizely Feature Experimentation and Optimizely Web Experimentation, but Optimizely Web Experimentation experiments include an extra layer: an experiment in Optimizely Web Experimentation can run on multiple Optimizely Web Experimentation pages. Every time a visitor activates a page within an Optimizely Web Experimentation experiment, Optimizely Web Experimentation counts an impression.
Sometimes, an experiment may have two pages that both target the same URL (this is common for single-page applications). When a visitor activates both pages during the same visit, this counts as two impressions.
Impressions are not counted for visitors bucketed into the holdback:
- In an experiment with the traffic allocation set to less than 100%, some visitors are in the holdback (
isLayerHoldbackis set to
true). These visitors do not see the experiment, and no impressions are counted for these visitors.
- In Optimizely Web Personalization with a holdback that’s greater than 0%, visitors in the holdback do not see a personalized experience. No impressions are counted for visitors bucketed into the campaign holdback.
Each time an Optimizely Web Experimentation experiment is activated, a decision request is sent. Decision requests look like this:
In the request payload, the decision attribute indicates the experiment that it applies to.
The following scenario has three multipliers:
- Pages (as defined in Optimizely Experimentation)
The Attic and Button company is experimenting on
www.atticandbutton.us. Consider a visitor who starts by visiting the Attic and Button homepage, where there are three experiments running. One of these experiments has two Optimizely Experimentation pages that both target the homepage:
- Homepage –
- Homepage –
- Homepage –
- Global page (a page defined in Optimizely Experimentation that targets all URLs of a website) –
Since four page activations occur when the visitor views the homepage, the visit counts as four impressions. Impressions are shown in green below.
If the visitor refreshes the page, another four impressions are counted. Attic and Button’s account usage now totals eight impressions.
Now, suppose that you are running a search algorithm experiment with Optimizely Feature Experimentation on the homepage too. When a visitor types a search term, the results are refreshed without reloading the page. The Optimizely Feature Experimentation SDK makes a decision for a variation every time a new search is done. This means that if a visitor searches for "shirts," changes their search to "denim shirts," then changes their search again to "button down shirts," another three impressions are counted. The total usage count is now 11 impressions.
Impression in Optimizely Feature Experimentation
See the developer documentation for information when impressions and decisions are counted for Optimizely Feature Experimentation.
Impression in Optimizely Full Stack Experimentation
In Optimizely Full Stack Experimentation, an impression is counted each time an experiment is activated and a decision event is sent:
- When the
optimizelyClientInstance.isFeatureEnabled()method is used.
- In iOS or Android SDK 1.x experiments, when
trueis passed with the
activateExperimentargument in the live variable getter methods. Returning Feature variables with 2.x SDKs does not send a decision event.
Impressions are only counted for visitors who are bucketed into a variation in an experiment. The
activate() call alone does not generate an impression.
Let us say that you are running an experiment on your homepage. In theory, every visitor to your homepage triggers the
activate() call. However, before an impression is created, the
activate() method does two things:
- Confirm that the visitor meets the specified audience conditions when setting up the experiment.
- Check the percentage of visitors you indicated should be included in the audience.
If the visitor meets the audience conditions, the
activate() call assigns a variation for the visitor. An impression is counted only for visitors who receive one of these variation assignments.
If the visitor does not meet the audience conditions, the
activate() call assigns a
NULL variation for the visitor, depending on the language. An impression is not counted for visitors who receive a
NULL variation assignment, which means these visitors do not count against your allotted number of impressions.
All rollouts are excluded from impression counts.
Verify impressions with results export
Optimizely Experimentation uses the server timestamp to calculate impressions, as opposed to the timestamp on the client device where the impression originated. Doing so lets you accurately verify impressions down to the experiment level.
You can use Optimizely Experimentation's Enriched Events Export to get a complete list of all monthly active users that occurred within a specific time period. You can then compare that information to your invoice, or determine whether any of your experiments are generating more monthly active users than they should be. To learn how to access that data, see Enriched events export.
Count impressions in billing
On June 23, 2020, Optimizely Experimentation began updated impression logic which is in effect automatically for your accounts. The new logic introduces fixed-interval bucketing which de-duplicates impressions received within fixed 5-second intervals for every user in an experiment. This de-duplicate is based on their received timestamp of the event. This de-duplication process is done by Optimizely Experimentation and does not require any action on your part to receive the benefit.
MAUs versus impressions (Legacy)
Optimizely Experimentation billed customers based on Monthly Active Users (MAUs) to simplify billing and volume tracking. Starting February 1, 2018, no new subscriptions use MAUs. However, some legacy subscriptions may still use MAUs.
To find out whether your subscription is billed based on MAUs go to Account Settings > Plan. Your MUVs are listed under Monthly Usage Information.