Enable email notifications in Optimizely Experimentation

  • Updated
  • Optimizely Feature Experimentation
  • Optimizely Web Experimentation
  • Optimizely Performance Edge
  • Optimizely Full Stack (Legacy)

Statistical significance and sample ratio mismatch detection notifications

To enable email notifications for when your project reaches statistical significance or Optimizely Experimentation detects a sample ratio mismatch, go to Profile > Notifications and select the corresponding projects for which you want to receive an email notification when a running experiment in that project reaches statistical significance or Optimizely detects a sample ratio mismatch.

Statistical significance notifications

When any of the experiment's metrics reach statistical significance, you receive an email providing enough context to help determine the urgency of any corresponding action. For example:

example statistical significance email

Optimizely Experimentation polls for statistical significance differently, depending on how long the experiment has been running:

  • 1-14 days – Polls every 6 hours.
  • 14-30 days – Polls every 24 hours.
  • 30-60 days – Polls every 36 hours.
  • 60-90 days – Polls every 48 hours.
  • 90-365 days – Polls every 96 hours.
  • 365+ days – Not monitored. 

You receive a notification for the metric's statistical significance value (measured between the control and each variation) that exceeds the project's statistical significance setting. Optimizely sends additional notifications for each new control and variation comparison occurrence that reaches statistical significance.

If you reset the experiment results and then a metric variation reaches statistical significant (again), this registers as a first-time occurrence, and Optimizely sends an email notification.

This feature does not impact result calculations, nor does it delay the real-time processing of your results page. The default view of the results page that uses unfiltered visitor segments and date ranges determines statistical significance.

Sample ratio mismatch notifications

If Optimizely Experimentation detects a sample ratio mismatch, you receive an email with the experiment's health status and information to review the experiment. For example: 

example sample ratio mismatch email notification

Optimizely Experimentation checks for sample ratio mismatches for running experiments one time a day. This means you only receive a notification once a day.

Impression notifications

  • Optimizely Full Stack (Legacy)

To subscribe to email notifications when your impressions used exceeds 75% of the impressions included in your plan, go to Profile > Notifications. Under Impression Usage, select Impression usage notifications to enable email communications.

enable impression usage notifications