Events from paused experiments, stopped variations, and archived experiments

  • Updated
Relevant products:
  • Optimizely Web Experimentation
  • Optimizely Performance Edge

There are times when it is necessary to pause an experiment. Before doing so, you should consider what happens to the user experience for visitors who were already bucketed into the experiment.

If you are using Optimizely Performance Edge, certain features described in this article behave differently than in Optimizely Web Experimentation. Optimizely Performance Edge is a lightweight experimentation product that delivers significantly faster performance than previous versions of Optimizely. It does this by relying on a streamlined "microsnippet" which limits the range of available features.

You see this notation edge-info.jpg whenever the text describes a feature that works differently in Optimizely Performance Edge than in Optimizely Web Experimentation.

Paused Experiments

If you pause a running experiment in Optimizely Web Experimentation, the variation code is removed from the snippet. This means the experiment and variation code is no longer available to run on the page because the code has been removed from the snippet.

Expected behavior of experiment:

  • New visitors are not evaluated for the experiment, meaning no new visitors are bucketed into the experiment once the experiment is paused.

  • Visitors already bucketed into the experiment while running see the native web page, with no experiment or variation code added.

Restarting Paused Experiments

If you restart a paused experiment, the experiment and variation code are both added back to the snippet. Visitors who were bucketed into the experiment while it was originally active again see the variation they saw before. Returning visitors are not re-bucketed into a new variation because their bucketing information, which is stored locally in their browsers, does not change if the experiment is paused.

Results Page

When an experiment is paused, the results page stops refreshing until the experiment is restarted.  Events scoped to the page that were triggered during the time the experiment was paused are still collected. However, they will not appear on the results page until the experiment is restarted.

While an experiment is paused, bucketing decisions are not collected for events scoped to the experiment. Therefore the results page may show a gap during the time period while the experiment was paused.

Network Events

Visitors who were already bucketed into the experiment can continue to trigger events while the experiment is paused. If you are testing your experiment and were bucketed into the experiment while it was running, then certain events (specifically, pageview, click events scoped to the page, and custom events) will continue to be triggered and visible in the Network tab.

Events scoped to the experiment are not triggered when an experiment is paused. Only Pageview events, click events scoped to a page, and custom events continue to be sent to Optimizely when an experiment is paused. These events are not visible on the results page. 

Stopped Variation edge-info.jpg

In some cases, you may need to stop a variation, but cannot stop the experiment. This is common when a winning variation is declared, and losing variations have to be stopped. When a variation is stopped, Optimizely removes both the variation code and the variation ID from the snippet, while leaving the experiment and any other variation information intact.

Expected behavior of an experiment

  • New visitors are evaluated by Optimizely Web Experimentation for the experiment and bucketed randomly into the remaining variations. This means no new visitors are bucketed into the stopped variation.

  • In Optimizely Web Experimentation, visitors who were already bucketed into the stopped variation see the native web page, with no experiment or variation code added.

  • edge-info.jpg In Optimizely Performance Edge, visitors who were already bucketed into the stopped variation are rebucketed into another variation in the experiment.

When you stop a variation, you cannot restart it.

Results Page

The results page will continue to collect data for the experiment. However, you should expect conversions for the stopped variation to decline.

Updating visitor bucketing information

Visitors who were bucketed into the stopped variation continue to attribute events to the results page until they go back to a URL where the experiment is active or running. Only then is bucketing information updated with the removal of the variation ID. At that point, events are no longer attributed to the variation ID on the results page.

Network Events

If you are testing a variation that has stopped, you may continue to see events triggered in the network tab. Open the network event, Navigate to the page where the experiment activates and open the network event. If the variation ID field is NULL, the bucketing information was updated.

Screen Shot 2018-01-08 at 2.52.38 PM.png

If an experiment with a stopped variation is duplicated, the duplicate version contains the stopped variation. To get around this, you can duplicate a stopped variation within an experiment. This lets code from the stopped variation run again without rebuilding it. However, the duplicated variation has a new ID, and none of the bucketed visitors from the stopped variation transfer to the duplicated copy.

Archived experiments

When an Optimizely Web Experimentation or Optimizely Performance Edge experiment is archived, the results page records a new stop date and time and ceases tracking events for the experiment. Refer to manage experiments and campaigns in Optimizely Web Experimentation for more information about archived experiments.

Related articles