- Optimizely Web Experimentation
By default, Optimizely Web Experimentation shows results for all visitors who enter your experiment. However, not all visitors behave like your average visitors. Optimizely Web Experimentation lets you filter your unarchived experiment results so you can see if certain groups of visitors behave differently from your visitors overall; this is called segmenting.
In Optimizely Web Experimentation, you can filter results with default segments that Optimizely captures automatically:
Segment by browser type.
Segment by campaign.
Segment by device type.
Segment by referrer.
Segment by source type.
Or, you can add custom attributes to your experiments and campaigns. Then, you can use these attributes for custom segmentation for a more granular view of important visitors to your business.
For example, imagine you run an experiment that tests a pop-up promotional offer. This generates a positive lift overall, but segmenting visitors on mobile devices is a statistically significant loss. Maybe the pop-up is disruptive on a mobile device or difficult to close. When you implement the change or run a similar test in the future, you might consider excluding mobile visitors based on what you have learned.
Segmenting results is one of the best ways to gain deep insight beyond the average visitor's behavior. It is a powerful way to step up your experimentation program.
Optimizely Web Experimentation currently does not offer segmentation by the audience.
Default segments: browser type, campaign, device type, referrer, and source type
Optimizely Web Experimentation captures several default attributes that you can use to segment your results page:
Browser type –
- Google Chrome
- Internet Explorer
Campaign – Visitors who enter the experiment with the parameter utm_campaign will capture the parameter value. Use this value to drill down into campaign keywords like "holiday_sale".
There is a 20-character limit for values. Longer values are truncated.
Device type –
- Desktop/laptop – Visitors rising from a desktop or laptop.
- All phones – Visitors using any mobile phone are shown in this segment. It is a combination of the iPhone and Other phone segments.
- iPhone – Apple iPhone visitors
- Other phones – Visitors using a mobile phone that is not an iPhone.
- All tablets – Visitors using any tablet will display in this segment It is a combination of the iPad and Other tablets segments.
- iPad * – Visitors using iPads
- Other tablets – Visitors using tablets that are not iPads.
- Other devices – Visitors using devices not identified as desktops, laptops, phones, or tablets will display in this segment.
- Desktop (deprecated) – Visitors using a desktop, laptop, or "other device" will display in this segment. Use the desktop/laptop segment to focus on visitors using traditional desktop browsers.
Referrer – Initial referrer or the originating URL (the URL of a page outside the domain of the snippet that the visitor navigated from before landing on a page where the snippet is implemented).
For example, if a visitor navigates to your domain by clicking on a link on https://www.facebook.com/search/top, events triggered during the session will have a referrer value of https://www.facebook.com/search/top.
Source type –
- Campaign – Includes visitors that arrive on a URL containing a 'utm_source', 'gclid', or 'otm_source' query parameter. If the URL contains one of these parameters, the visitor will count as "Campaign" traffic, even if they arrived through a search.
- Direct – Includes all visitors who do not have any external referrer in their URL.
- Referral – Includes all visitors from another URL that do not count as Campaign.
- Search – Includes all visitors who arrived through search.
*Apple changed the user agent for iPad in iOS 13 such that it will display as desktop/laptop traffic in Optimizely Web Experimentation and other analytics tools. This may result in an increase in desktop/laptop traffic and a decrease in iPad traffic when segmenting your results.
The "Source" value is based on the
document.referrer value in the browser, except "Campaign." Visitors' "Source" type value may change as they navigate your site.
For example, a checkout flow that includes a step where the visitor goes to a third-party site or different subdomain and then returns to your site. Visitors would have an original "Source" value (campaign, referral, or direct) until they leave the site. When the visitor returns, they will have a "referral" value because Optimizely Web Experimentation detects that they went to another site.
You may see the same visitor's conversions in segments for different "Source" values on your Results page. For example, conversions that the visitor made before leaving the site will show up in "campaign" or "direct." And conversions triggered after the visitor returns will show up in "referral" without affecting the earlier conversions.
In Optimizely Web Experimentation, you can create custom attributes and use them for custom segmentation (with more granularity) on the results page. Custom attributes describe key characteristics of your different visitors.
For example, you might create a custom attribute that targets by "plan type," which includes the values: basic, plus, and premium. You can use these to segment your results and identify when visitors with a basic plan behave differently from visitors with a premium plan.
Here is how to create a custom attribute in Optimizely Web Experimentation. Use them to segment your results page in the exact same way as default segments.
Segment the Results page
In Optimizely Web Experimentation, you can segment your entire results page. Segmenting results helps you get more out of your data by generating valuable insights about your visitors.
Under the Segment drop-down list, you will find default segments and custom segments for any custom attributes assigned to at least one visitor in the experiment.
Go to your results page.
Click Segment and select one or more attribute values from the drop-down list.
When selecting multiple attribute values, choose Match All Values or Match Any Value.
Match All Values and Match Any Value
When you select more than one attribute value from the segmentation menu, the results page can display visitors who match all values or visitors who match one or more values. The radio buttons control this behavior labeled Match All Values and Match Any Value.
For example, you have implemented two custom attributes to track a visitor’s location and plan type. If you want to see how visitors in the US with a Gold plan responded to your variations, your segmentation configuration will look like this:
Match All Values
Location – US
Plan Type – Gold
If you select Match Any Value instead, the results page will display data for visitors who are in the US combined with data for visitors who have a Gold plan.
You cannot make any statistically rigorous claims from segmented results because Optimizely does not enforce any multiple-hypothesis test correction in segmented results. If you look at multiple segments of the same experiment, you have a multiple comparisons problem. Thus, segmented results should only be used for exploratory data analysis, and the statistical significance claims should be disregarded. It is recommended to use segmented results as inspiration for future iterations of tests. See Segmentation and statistical significance for more.
How visitors are counted in segments
For information on how Optimizely counts conversions when segmenting, see: