- Segment your Results page to see how different groups of visitors behave
- Identify if certain groups of visitors behave differently from the your visitors overall
- Use default segments or create custom segments using attributes
By default, Optimizely shows results for all visitors who enter your experiment. However, not all visitors behave like your average visitors. Optimizely lets you filter your results so you can see if certain groups of visitors behave differently from your visitors overall; this is called segmenting.
In Optimizely, you can always filter results with default segments that we capture automatically:
Segment by browser type
Segment by campaign
Segment by device type
Segment by referrer
Segment by source type
Or, you can add custom attributes to your experiments and campaigns. Then, you can use these attributes for custom segmentation for a more granular view of visitors who are important to your business.
For example, imagine you run an experiment that tests a pop-up promotional offer. This generates a positive lift overall, but when you segment for visitors on mobile devices, it's a statistically significant loss. Maybe the pop-up is very disruptive on a mobile device, or it's difficult to close. When you implement the change or run a similar test in the future, you might consider excluding mobile visitors based on what you've learned.
Segmenting results is one of the best ways to gain deep insight beyond the average visitor's behavior. It's a powerful way to step up your experimentation program.
Optimizely currently does not offer segmentation by audience.
Default segments: browser type, campaign, device type, referrer, and source type
Optimizely captures several default attributes that you can always use to segment your Results page:
Visitors who enter the experiment with the parameter utm_campaign will capture the parameter value. Use this to value to drill down into campaign keywords like “holiday_sale". There is a 20-character limit for values; longer values will be truncated.
Desktop / laptop
All phones: Visitors using any mobile phone will appear in this segment. It's a combination of the iPhone and Other phones segments.
All tablets: Visitors using any tablet will appear in this segment. It's a combination of the iPad and Other tablets segments.
Other devices: Visitors using any device not identified as desktop, laptop, phone, or tablet will appear in this segment.
Desktop (deprecated): Visitors using a desktop, laptop, or "other device" will appear in this segment. Use the desktop / laptop segment to focus on visitors using traditional desktop browsers.
Initial referrer or the originating URL (the URL of a page outside the domain of the snippet that the visitor navigated from before landing on a page where the snippet is implemented). For example, if a visitor navigates to your domain by clicking a link on https://www.facebook.com/search/top/, then all events triggered during the session will have a referrer value of "https://www.facebook.com/search/top/".
Campaign: Includes visitors that arrive on a URL containing a 'utm_campaign,' 'utm_source,' 'gclid,' or 'otm_source' query parameter. If the URL contains one of these parameters, the visitor will count as "Campaign" traffic even if they arrived through search.
Direct: Includes all visitors who do not have any external referrer in their URL.
Referral: Includes all visitors that come from another URL that doesn't count as Campaign.
Search: Includes all visitors who arrived through search.
*Apple changed the user agent for iPad in iOS 13 such that it will appear as desktop / laptop traffic in Optimizely and other analytics tools. This may result in an increase in desktop / laptop traffic and a decrease in iPad traffic when segmenting your results.
The "Source" value is based on the
document.referrer value in the browser, with the exception of "Campaign." This means a visitor's "Source" type value may change as they navigate your site.
For example, imagine a checkout flow that includes a step where the visitor goes to a third-party site or different subdomain, then returns to your site. The visitor would have an original "Source" value (campaign, referral, or direct) until they leave the site. Once the visitor returns, they'll have a "referral" value because Optimizely detects that they went to another site.
On your Results page, you may see the same visitor's conversions in segments for different "Source" values. For example, conversions that the visitor made before leaving the site will show up in "campaign" or "direct." And conversions triggered after the visitor returns will show up in "referral," without affecting the earlier conversions.
In Optimizely, customers with Grow, Accelerate, and Scale Plans can create custom attributes and use them for custom segmentation (with more granularity) on the Results page. Custom attributes describe key characteristics of your different visitors.
For example, you might create a custom attribute that targets by "plan type," which includes the values basic, plus, and premium. You can use these to segment your results and identify when visitors with a basic plan behave differently from visitors with a premium plan.
Segment the Results page
In Optimizely, you can segment your entire Results page. Segmenting results helps you get more out of your data by generating valuable insights about your visitors.
Under the Segment dropdown, you'll find default segments as well as custom segments for any custom attributes that have been assigned to at least one visitor in the experiment.
Navigate to your Results page
Click Segment and select one or more attribute values from the dropdown
When selecting multiple attribute values, choose Match All Values or Match Any Value
When you select more than one attribute value from the segmentation menu, the results page can either display visitors who match all values, or visitors who match one or more values. This behavior is controlled by the radio buttons labeled Match All Values and Match Any Value.
For example, let’s say you have implemented two custom attributes to track a visitor’s location and a visitor’s plan type. If you want to see how visitors in the US with a Gold plan responded to your variations, your segmentation configuration would look like this:
Match All Values
Plan Type: Gold
If you select Match Any Value instead, the results page will display data for visitors who are in the US combined with data for visitors who have a Gold plan.
You cannot make any statistically rigorous claims from segmented results because we do not enforce any multiple hypothesis test correction in segmented results. If you look at multiple segments of the same experiment, you have a multiple comparisons problem. Thus, segmented results should only be used for exploratory data analysis and the statistical significance claims disregarded. It's recommended to use segmented results as inspiration for future iterations of tests.
How visitors are counted in segments
When segmenting results, a visitor who belongs to more than one segment will be counted in every segment they belong to.
Below are two examples that illustrate how visitors and conversions are counted.
Imagine that a visitor enters an experiment with the basic plan type (attribute name: "plan type", value: "basic") and triggers a conversion event. How is that visitor counted?
If you segment the Results page for the basic plan type, the visitor is counted once and the conversion is counted once.
Now, imagine that same visitor upgrades to a premium plan and enters the experiment again in a new session. They trigger a second conversion event. How are they counted?
If you segment the Results page for the premium plan type, the visitor and conversion are counted once. If you segment for the basic plan type, the visitor is counted and the initial conversion is also counted once.
In the overall view of the experiment, only one visitor and one conversion event is counted, since the visitor has the same unique user ID.
Imagine that a visitor enters the experiment with the basic plan type but doesn't convert. The same visitor comes back to the experiment with a premium plan and now triggers a conversion. How is that visitor counted?
If you segment the Results page for the basic plan type, you'll see one visitor but no conversions.
If you segment the Results page for the premium plan, you'll see one visitor and one conversion.
In the overall view of the experiment, you'll see one visitor and one conversion event.