Optimizely Web Personalization Campaign Results page

  • Updated
  • Optimizely Web Personalization

The Campaign Results page lets you track how well your campaigns perform, examine different experiences and audiences for a detailed view of your visitors' behavior, and take action.

  • A session starts when a visitor lands on your site and ends after 30 minutes of inactivity. A new session is created if there is new activity.
  • By default, personalization maintains a 5% holdback:
    • 95% of visitors in a campaign are shown a personalized experience.
    • 5% holdback of visitors who will see the original. 

The Campaign Results page helps you track how well your campaigns perform. You use it to measure the success of each experience you deliver and explore your visitors' behavior. You can get to the Results page in the following ways:

  • From the Experiments dashboard, click Results.
    Reports-1.png

  • From the Manage Campaign dashboard, click View Results.
    Reports-2.png

Results in Optimizely Web Personalization are in local time, according to the time zone set on your machine. 

Campaign Summary

The Campaign Summary helps you track your campaign's overall impact. In the sidebar, you can view when the campaign was last published and how many days it is running.

Overall Improvement

The Overall Improvement module provides a high-level view of your campaign's progress.

campaign-summary.png

By default, the module shows you the improvement in the campaign's primary metric with a confidence interval. The improvement measures the relative difference between personalized sessions and the holdback (original) experience. It also tells you about the reach of campaign reach: the number of personalized sessions delivered to visitors and the impact of that metric per session.

Use the dropdown menu to toggle between metrics that are attached to your campaign.

pers-results-change-metrics.png

Or, use the calendar dropdown to select a different date range and click Apply.

pers-results-filter-date.png

Campaign Reach

The Campaign Reach module gives you a sense of how many personalized experiences were delivered, by session.

pers-campaign-reach.png

  • Page reach – Shows the total number of sessions that occurred on pages that were targeted by your campaign.
  • Audience reach – Shows the percentage of sessions that were delivered to one or more audiences in the campaign.
  • Personalized sessions – Shows the number of sessions where a visitor saw a personalized experience.
  • Holdback sessions – Shows the number of sessions where a visitor saw the original experience because they're bucketed into the holdback by chance.

Experience Breakdown

The Experience Breakdown provides a visual summary of the impact of the campaign, divided by audience.

pers-exp-breakdown.png

The example above shows four audiences: Existing Customers, Media Vertical, Strategic Accounts, and High Tech Vertical

The width (x-axis) of each bar shows the reach of a particular audience. For example, the Existing Customers audience makes up 38.39% of total sessions in the campaign. All audiences in a campaign add up to 100%.

The height (y-axis) of each bar shows the change in conversion rate. The Existing Customers audience saw 20.99% lift in the primary metric, conversion rate, in this case. For conversion events such as clicks or pageviews, the y-axis shows the conversion rate. Revenue is displayed as revenue per visitor.

The color of the bars shows whether the change in conversion rate is statistically significant:

  • A green bar represents a statistically significant lift
  • A red bar represents a statistically significant loss
  • A gray bar represents a change that hasn't reached statistical significance

To see more granular data about the primary metric by audience, click More Details.

pers-more-details.png

The Impact column shows the difference in conversions in a personalized experience, compared to the holdback. It's broken down by audience.

Audience Summary

The audience summary helps you track the improvement generated by the personalized experience for each audience, broken down by metric.

Click to select an audience and view a summary.

pers-audience-summary.png

Once you enter an audience summary, you can see how each metric performed for that audience.

The summary will always show your primary metric first (in the example below, the Requested Demo event).

pers-audience-summary-primary-metric.png

To view a graph of conversions in a specific metric for a particular audience, click View graph.

pers-graph.png

Segment results

Optimizely Web Personalization lets you segment your results by attributes like browser, device, campaign, and source so you can drill down into the behavior of certain groups of visitors.

Segmenting your results gives you valuable insights about different groups of visitors instead of analyzing the results across your entire visitor base. 

Metrics definitions

  • Sessions – A session starts when a visitor lands on your site, continues as they go between pages, and ends after 30 minutes of inactivity—at which point the next visit creates a new session.
    Formula:
    • Absolute number
  • Personalized sessions – The number of sessions where a visitor was shown a personalized experience. This excludes visitors who either (1) do not qualify for any audience in the campaign or (2) qualify for an audience but are randomly bucketed into the holdback (control group).
    Formula:
    • Sum the treatment size over all Audiences
  • Reach – The percentage of total sessions that qualified for an audience (regardless of whether those visits were personalized).
    Formula:
    • Number of sessions for an audience / Total sessions
  • Conversions – The number of sessions in which a visitor from any campaign audience converted on this event.
    Formula:
    • Absolute number
  • Conversion Rate – The percentage of sessions in which a visitor from any campaign audience converted on this event.
    Formula:
    • The number of sessions where conversions took place for that audience, or for total sessions.
  • Holdback – Every campaign has a "holdback," which is like the control of an A/B test.
    You can adjust the holdback for each campaign.
    Formula:
    • By default, Optimizely Web Personalization shows 95% of visitors in a campaign a personalized experience and holds back 5% who will see the original.
  • Confidence Interval – The confidence interval gives a range of values in which the conversion rate for a particular experience actually lies.
    As you collect more data, the confidence interval will narrow. After your variation reaches statistical significance, the confidence interval lies entirely above or below 0.
    Formula:
  • Improvement – The relative improvement of this event's conversion rate for personalized sessions, compared to the holdback.
    Formula:
    • (Event Conversion Rate for Personalized visits - Event Conversion rate for Control visits) / (Event Conversion Rate for Control visits)
  • Improvements for an event for each audience – The % improvement in an event's conversion rate, comparing a personalized audience to the control.
    Formula:
    • (Event Conversion Rate for Audience's Personalized visits - Event Conversion rate for Audience control visits) / (Event Conversion Rate for Audience control visits)
  • Total Improvement in Primary Metric – The % improvement  of the Primary Metric across all sessions in the campaign (regardless of whether they fell into an audience).
    Formula:
    • (Conversion rate for sessions in the treatment - conversion rate of sessions in control) / conversion rate of sessions in control

Add or remove metrics

Existing events and tags are tracked, so you can add them back if you want. However, if you add more than five events, it may take longer to reach statistical significance.

  1. Click Manage Metrics.
    pers-results-manage-metrics.png
  2. Click Add (+) to add metrics or drag metrics to remove them.
    pers-results-manage-metrics-2.png

Set currency symbols

You can change the currency symbols displayed for revenue metrics with the Currency Picker, located in Project SettingsAdvanced:

Optimizely_changeCurrency.png

Note that changing the currency symbol does not convert revenue values based on an exchange rate, it only changes the symbol displayed. So a revenue value displayed as $100.00 would become ¥100 after picking ¥, regardless of the current exchange rate between USD and JPY. Learn more about tracking revenue in Optimizely Web Personalization.

The currency symbol setting you select is remembered for the sessions on your current browser. This means you need to set the symbol again when logging in from a new device or browser.

How quickly do results update?

Results are typically available within 5 to 10 minutes of Optimizely Web Personalization receiving the data. See data freshness.

Reset or delete results

To remove visitors and conversions from your Results page, click Reset Results on the results page Campaign sidebar.

pers-reset-results.png

At this time, clicking Reset Results resets your results to the state of the last hourly refresh of your results. So even after you click reset, you might see conversions and visitors who were part of your Campaign from this timeframe.