Enable email notifications in Optimizely Experimentation

Statistical significance and sample ratio mismatch detection notifications

To enable email notifications for when your project reaches statistical significance or Optimizely Experimentation detects a sample ratio mismatch, go to Profile > Notifications and select the corresponding projects for which you want to receive an email notification when a running experiment in that project reaches statistical significance or Optimizely detects a sample ratio mismatch.

Statistical significance notifications

When any of the experiment's metrics reach statistical significance, you receive an email providing enough context to help determine the urgency of any corresponding action. For example:

Optimizely Experimentation polls for statistical significance differently, depending on how long the experiment has been running:

• 1-14 days – Polls every 6 hours.
• 14-30 days – Polls every 24 hours.
• 30-60 days – Polls every 36 hours.
• 60-90 days – Polls every 48 hours.
• 90-365 days – Polls every 96 hours.
• 365+ days – Not monitored. 

You receive a notification for the metric's statistical significance value (measured between the control and each variation) that exceeds the project's statistical significance setting. Optimizely sends additional notifications for each new control and variation comparison occurrence that reaches statistical significance.

If you reset the experiment results and then a metric variation reaches statistical significant (again), this registers as a first-time occurrence, and Optimizely sends an email notification.

Sample ratio mismatch notifications

If Optimizely Experimentation detects a sample ratio mismatch, you receive an email with the experiment's health status and information to review the experiment. For example: 

Optimizely Experimentation checks for sample ratio mismatches for running experiments one time a day. This means you only receive a notification once a day.

Impression notifications

To subscribe to email notifications when your impressions used exceeds 75% of the impressions included in your plan, go to Profile > Notifications. Under Impression Usage, select Impression usage notifications to enable email communications.