Guardrail alerts

The New A/B Results page is currently in beta. Contact your Customer Success Manager or sign up for the beta on Optimizely.com.

Guardrail alerts let you configure metric alerts that protect business-important metrics during an experiment. Each alert watches a metric you choose and notifies you when a variation moves that metric past a threshold you define, relative to the baseline. Optimizely sends the alert so you can keep an eye on the metric without constantly returning to the results page.

You can use guardrail alerts to:

• Catch regressions early and decide whether to continue, pause, or adjust the experiment.
• Guard metrics such as revenue, checkout completion, signup rate, session time, or retention.
• Reduce the manual monitoring of running experiments.
• Give your team an early warning when a variation harms a critical key performance indicator (KPI).

How guardrail alerts work

You configure a guardrail alert on a specific metric. For that metric, every variation has an improvement value relative to the baseline, and the alert watches those values.

You define the threshold as a percentage change relative to the baseline. Its sign sets the direction: a negative threshold watches for a drop, and a positive threshold watches for an increase. Optimizely compares the threshold against each variation's improvement and triggers the alert when any variation breaches it.

Optimizely measures improvement relative to the baseline. For example, a baseline of 10% that rises to 11% is a 10% relative improvement.

Consider a metric whose variations show improvements of +20% and +3%, with a threshold of -5%. Neither variation breaches the threshold, so no alert fires. If another variation drops to -6%, it breaches the -5% threshold, and Optimizely notifies you. This protects a metric such as revenue per user. Optimizely notifies you if revenue drops in any variation, so you do not have to check the results page yourself.

The following conditions let you configure how sensitive the alert is:

• Minimum visitor count – Set a minimum visitor count so the alert evaluates only after each variation and the baseline reach that count. Early in an experiment, when the number of visitors is low, metrics are volatile and the improvement swings widely. A higher minimum visitor count prevents alerts before the signal is steady.
• Statistical significance (optional) – Add statistical significance as a third condition. When you turn it on, an alert fires only after a variation breaches the threshold, reaches the minimum visitor count, and reaches statistical significance.

Configure guardrail alerts

You can configure guardrail alerts on a per-metric level. Complete the following steps on the results page:

1. Under Advanced Settings, toggle Set alerts on for the metric you want to monitor.

   

2. In Alert when threshold is breached, enter the percentage change that triggers the alert and select a positive or negative value.
3. In Alert only if users count is at least, enter the minimum visitor count. Optimizely sends an alert only after the baseline and every variation reach this count. For example, set 10,000 to wait until the estimate is less volatile.
4. In Notify, choose the users who receive the alert through email. By default, the field lists the users in your instance. You can leave the field blank and subscribe to receive alerts through Slack.
5. (Optional) Select Alert only when Statsig is reached. The alert then fires only when a variation breaches the threshold, meets the visitor count, and reaches statistical significance.
6. Click Save.

How you receive alerts

Optimizely sends guardrail alerts two ways: by Email and through Slack.

Email alerts

Email alerts go only to the users you select in the Notify field when you configure the guardrail alert. The following is an example of an email alert:

Slack alerts

To receive guardrail alerts in Slack, complete the following steps:

1. Go to More (…) > Tools > Apps.

2. Add the Optimizely app to Slack.

   

3. Click Login to Experimentation and sign in. After you sign in, the following Slack commands are available:
   • /subscribe – Subscribe the current channel to a project.
   • /unsubscribe – Unsubscribe from a project.
   • /unsubscribe-all – Unsubscribe the channel from all project notifications.
   • /show-subscribed-projects – List the experimentation projects the channel subscribes to.

   • /optimizely-help – Open the help prompt with command guidance.

     

4. Open the Slack channel where you want to receive alerts. To invite the Optimizely app, type @Optimizely, click Send, then click Add them.

   

5. Type /subscribe and click Send. Optimizely then posts every guardrail alert from every experiment in that project to the channel.

   

6. Use the Project Selection drop-down list to choose the project whose alerts post to the channel.

   

The Slack channel now receives alerts for every metric with guardrail alerts enabled in your selected project. The following is an example of a Slack alert:

Alert frequency

Optimizely checks guardrail alerts on the following schedule:

• Every six hours for the first 15 days of the experiment.
• Once a day from day 16 through day 30.
• Checks stop after day 30.