Impression in Optimizely Experimentation

This article describes how impressions work at Optimizely Experimentation, similar to that used by online media, but the nature of our technology introduces some key differences.

Impression in Optimizely Web Experimentation

In Optimizely Web Experimentation, an impression is counted for every Optimizely Web Experimentation page each time a visitor sees an Optimizely Web Experimentation experience as the result of a test or personalization campaign.

Impressions are the unit of measurement for usage in Optimizely Feature Experimentation and Optimizely Web Experimentation, but Optimizely Web Experimentation experiments include an extra layer: an experiment in Optimizely Web Experimentation can run on multiple Optimizely Web Experimentation pages. Every time a visitor activates a page within an Optimizely Web Experimentation experiment, Optimizely Web Experimentation counts an impression.

Sometimes, an experiment may have two pages that target the same URL (this is common for single-page applications). When a visitor activates both pages during the same visit, this counts as two impressions.

Impressions are not counted for visitors bucketed into the holdback:

• In an experiment with the traffic allocation set to less than 100%, some visitors are in the holdback (isLayerHoldback is set to true). These visitors do not see the experiment, and no impressions are counted for these visitors.
• In Optimizely Personalization with a holdback that’s greater than 0%, visitors in the holdback do not see a personalized experience. No impressions are counted for visitors bucketed into the campaign holdback.

Decision event

A decision request is sent each time an Optimizely Web Experimentation experiment is activated. Decision requests look like this:

In the request payload, the decision attribute indicates the experiment that it applies to.

Example

The following scenario has three multipliers:

• Experiments
• Pages (as defined in Optimizely Experimentation)
• Pageviews

The Attic and Button company is experimenting on www.atticandbutton.us. Consider a visitor who starts by visiting the Attic and Button homepage, where there are three experiments running. One of these experiments has two Optimizely Experimentation pages that target the homepage:

Experiment 1

• Homepage – https://www.atticandbutton.us/ (simple match)

Experiment 2

• Homepage – https://www.atticandbutton.us/ (simple match)

Experiment 3

• Homepage – https://www.atticandbutton.us/ (simple match)
• Global page (a page defined in Optimizely Experimentation that targets URLs of a website) – https://www.atticandbutton.us/ (substring match)

The visit counts as four impressions because four page activations occur when the visitor views the homepage. Impressions are shown in green below.

If the visitor refreshes the page, another four impressions are counted. Attic and Button’s account usage now totals eight impressions.

Now, suppose you are also running a search algorithm experiment with Optimizely Feature Experimentation on the homepage. When a visitor types a search term, the results are refreshed without reloading the page. The Optimizely Feature Experimentation SDK makes a decision for a variation every time a search is done. This means that if a visitor searches for "shirts," changes their search to "denim shirts," then changes their search again to "button down shirts," another three impressions are counted. The total usage count is now 11 impressions.

Impression in Optimizely Feature Experimentation

See the developer documentation for information on when impressions and decisions are counted for Optimizely Feature Experimentation.

Impression in Optimizely Full Stack Experimentation

In Optimizely Full Stack Experimentation, an impression is counted each time an experiment is activated and a decision event is sent:

• When the optimizelyClientInstance.activate() or optimizelyClientInstance.isFeatureEnabled() method is used.
• In iOS or Android SDK version 1.X experiments, when true is passed with the activateExperiment argument in the live variable getter methods. Returning Feature variables with version 2.X+ SDKs do not send a decision event.

Impressions are only counted for visitors who are bucketed into a variation in an experiment. The activate() call alone does not generate an impression.

Let us say that you are running an experiment on your homepage. In theory, every visitor to your homepage triggers the activate() call. However, before an impression is created, the activate() method does two things:

• Confirm that the visitor meets the specified audience conditions when setting up the experiment.
• Check the percentage of visitors you indicated should be included in the audience.

If the visitor meets the audience conditions, the activate() call assigns a variation for the visitor. An impression is counted only for visitors receiving one variation assignment.

If the visitor does not meet the audience conditions, the activate() call assigns a nil or NULL variation for the visitor, depending on the language. An impression is not counted for visitors who receive a nil or NULL variation assignment means these visitors do not count against your allotted number of impressions.

All rollouts are excluded from impression counts.

Verify impressions with results export

Optimizely Experimentation uses the server timestamp to calculate impressions instead of the timestamp on the client device where the impression originated. Doing so lets you accurately verify impressions down to the experiment level.

You can use Optimizely Experimentation's Experimentation Events Export to get a complete list of monthly active users within a specific period. You can then compare that information to your invoice or determine whether your experiments generate more monthly active users than they should. To learn how to access that data, see Experimentation Events Export.

Count impressions in billing

On June 23, 2020, Optimizely Experimentation began updating impression logic which is in effect automatically for your accounts. The logic introduces fixed-interval bucketing, which de-duplicates impressions received within fixed 5-second intervals for every user in an experiment. This de-duplicate is based on the received timestamp of the event. This de-duplication process is done by Optimizely Experimentation and does not require any action on your part to receive the benefit. 

View Impression usage

View MAU usage if you are billed by MAUs.

If you are on Impressions billing, to view your Impression Usage Summary in the Optimizely Application, go to Account Settings > Usage.