Segment your results on Optimizely Web Experimentation

  • Updated

Relevant products:

  • Optimizely Web Experimentation

This topic describes how to:

  • Segment your Results page to see how different groups of visitors behave
  • Identify if certain groups of visitors behave differently from your visitors overall
  • Use default segments or create custom segments using attributes

By default, Optimizely Web Experimentation shows results for all visitors who enter your experiment. However, not all visitors behave like your average visitors. Optimizely Web Experimentation lets you filter your unarchived experiment results so you can see if certain groups of visitors behave differently from your visitors overall; this is called segmenting. 

In Optimizely Web Experimentation, you can always filter results with default segments that we capture automatically:

  • Segment by browser type

  • Segment by campaign

  • Segment by device type

  • Segment by referrer

  • Segment by source type

Or, you can add custom attributes to your experiments and campaigns. Then, you can use these attributes for custom segmentation for a more granular view of important visitors to your business.

For example, imagine you run an experiment that tests a pop-up promotional offer. This generates a positive lift overall, but when you segment visitors on mobile devices, it is a statistically significant loss. Maybe the pop-up is very disruptive on a mobile device or difficult to close. When you implement the change or run a similar test in the future, you might consider excluding mobile visitors based on what you have learned.

Segmenting results is one of the best ways to gain deep insight beyond the average visitor's behavior. It is a powerful way to step up your experimentation program.

Optimizely Web Experimentation currently does not offer segmentation by the audience.

Default segments: browser type, campaign, device type, referrer, and source type

Optimizely Web Experimentation captures several default attributes that you can always use to segment your Results page:

Browser type
  • Firefox
  • Google Chrome
  • Internet Explorer
  • Opera
  • Safari
  • Unknown
Campaign

Visitors who enter the experiment with the parameter utm_campaign will capture the parameter value. Use this value to drill down into campaign keywords like "holiday_sale".

There is a 20-character limit for values; longer values will be truncated.
Device type
  • Desktop/laptop
  • All phones - Visitors using any mobile phone will appear in this segment. It is a combination of the iPhone and Other phone segments.
  • iPhone
  • Other phones
  • All tablets - Visitors using any tablet will appear in this segment It is a combination of the iPad and Other tablets segments.
  • iPad *
  • Other devices - Visitors using devices not identified as desktops, laptops, phone, or tablets will appear in this segment.
  • Desktop (deprecated) - Visitors using desktop, laptop, or "other device" will appear in this segment. Use desktop/laptop segment to focus on visitors using tradition desktop browsers.
Referrer

Initial referrer or the originating URL (the URL of a page outside the domain of the snippet that the visitor navigated from before landing on a page where the snippet is implemented).

For example, if a visitor navigates to your domain by click on a link on https://www.facebook.com/search/top, all events triggered during the session will hav a referrer value of https://www.facebook.com/search/top. 
Source type

Campaign - Includes visitors that arrive on a URL containing a 'utm_source', 'gclid', or 'otm_source' query parameter. If the URL contains one of these parameters, the visitor will count as "Campaign" traffic, even if they arrived through a search.

Direct - Includes all visitors who do not have any external referrer in their URL.

Referral - Includes all visitors from another URL that do not count as Campaign.

Search - Includes all visitors who arrived through search.

*Apple changed the user agent for iPad in iOS 13 such that it will appear as desktop/laptop traffic in Optimizely Web Experimentation and other analytics tools. This may result in an increase in desktop/laptop traffic and a decrease in iPad traffic when segmenting your results.

The "Source" value is based on the document.referrer value in the browser, except "Campaign." Visitors' "Source" type value may change as they navigate your site.

For example, imagine a checkout flow that includes a step where the visitor goes to a third-party site or different subdomain, then returns to your site. Visitors would have an original "Source" value (campaign, referral, or direct) until they leave the site. Once the visitor returns, they'll have a "referral" value because Optimizely Web Experimentation detects that they went to another site.

You may see the same visitor's conversions in segments for different "Source" values on your Results page. For example, conversions that the visitor made before leaving the site will show up in "campaign" or "direct." And conversions triggered after the visitor returns will show up in "referral" without affecting the earlier conversions.

Custom segmentation

In Optimizely Web Experimentation, customers with Grow, Accelerate, and Scale Plans can create custom attributes and use them for custom segmentation (with more granularity) on the Results page. Custom attributes describe key characteristics of your different visitors.

For example, you might create a custom attribute that targets by "plan type," which includes the values basic, plus, and premium. You can use these to segment your results and identify when visitors with a basic plan behave differently from visitors with a premium plan.

Here is how to create a custom attribute in Optimizely Web Experimentation. Use them to segment your Results page in the exact same way as default segments.

Segment the Results page

In Optimizely Web Experimentation, you can segment your entire Results page. Segmenting results helps you get more out of your data by generating valuable insights about your visitors.

Under the Segment dropdown, you will find default segments and custom segments for any custom attributes assigned to at least one visitor in the experiment.

segment-results-web.png

  1. Navigate to your Results page.

  2. Click Segment and select one or more attribute values from the dropdown.

  3. When selecting multiple attribute values, choose Match All Values or Match Any Value.

  4. Click Apply.

Match All Values and Match Any Value

When you select more than one attribute value from the segmentation menu, the results page can display visitors who match all values or visitors who match one or more values. The radio buttons control this behavior labeled Match All Values and Match Any Value.

For example, let us say you have implemented two custom attributes to track a visitor’s location and plan type. If you want to see how visitors in the US with a Gold plan responded to your variations, your segmentation configuration will look like this:

Match All Values

  • Location – US

  • Plan Type – Gold

If you select Match Any Value instead, the results page will display data for visitors who are in the US combined with data for visitors who have a Gold plan.

You cannot make any statistically rigorous claims from segmented results because we do not enforce any multiple hypothesis test correction in segmented results. If you look at multiple segments of the same experiment, you have a multiple comparisons problem. Thus, segmented results should only be used for exploratory data analysis, and the statistical significance claims should be disregarded. It is recommended to use segmented results as inspiration for future iterations of tests.

How visitors are counted in segments

When segmenting results, a visitor who belongs to more than one segment will be counted in every segment they belong to.

Below are two examples that illustrate how visitors and conversions are counted.

Imagine that a visitor enters an experiment with the basic plan type (attribute name: "plan type," value: "basic") and triggers a conversion event. How is that visitor counted?

  • If you segment the Results page for the basic plan type, the visitor is counted once, and the conversion is counted once. 

Now, imagine that same visitor upgrades to a premium plan and enters the experiment again in a new session. They trigger a second conversion event. How are they counted?

  • If you segment the Results page for the premium plan type, the visitor and conversion are counted once. If you segment for the basic plan type, the visitor is counted, and the initial conversion is also counted once. 

  • In the overall view of the experiment, only one visitor and one conversion event are counted since the visitor has the same unique user ID.

Imagine that a visitor enters the experiment with the basic plan type but does not convert. The same visitor comes back to the experiment with a premium plan and now triggers a conversion. How is that visitor counted?

  • If you segment the Results page for the basic plan type, you will see one visitor but no conversions.

  • If you segment the Results page for the premium plan, you will see one visitor and one conversion.

  • In the overall view of the experiment, you will see one visitor and one conversion event.