Relevant products:
- Optimizely Web Experimentation
- Optimizely Performance Edge
This topic describes how to:
- Identify which bots are filtered out by Optimizely Web
- Learn how bots affect experiment results, monthly unique visitors (MUVs), and billing
Optimizely Performance Edge is a lightweight experimentation product that delivers significantly faster performance than previous versions of Optimizely. It does this by relying on a streamlined "microsnippet" which limits the range of available features.
Optimizely Performance Edge and Optimizely Web filter bots and spiders the same way, so all information contained in this article applies to experiments created with either version of Optimizely.
Internet bots and spiders are software applications that run repetitive, automated tasks over the Internet. Optimizely Web uses the IAB/ABC international spiders and bots list to classify non-human traffic and filter it out of your results. The IAB/ABC list is an actively maintained record of user agents used by known spiders and bots. The list is updated on a monthly basis.
Bots and spiders don't behave like real users. They typically run repetitive, automated tasks over your website or app at a much higher rate than real users, with the purpose of crawling your content and analyzing it. As a result, bots and spiders usually inflate the volume of traffic to your site and may skew your conversion metrics.
Removing bot and spider traffic from your results generally decreases the volume of traffic, such as the total number of visitors or sessions, and makes your measurements more accurate.
Optimizely reads the
User-Agent header for each request and determines whether it matches with an entry in the IAB/ABC list. If there is a match, Optimizely classifies the request as non-human and automatically excludes it from the results.
Bot and spider filtering relies on the client’s
User-Agent header to work. For web traffic, the User-Agent header is normally sent with every request, so Optimizely Web customers will automatically benefit from it.
Yes. This capability is available for Node and JavaScript, starting with the JavaScript version 2.1.0 SDK. If you are a Full Stack user who wants to take advantage of bot and spider filtering, contact Optimizely and ask to be white-listed.
Optimizely Full Stack
only filters results automatically for
JavaScript SDK network requests for events from the browser. In this case, the User-Agent request header contains the same value in
navigator.userAgent
(the visitor's browser's user agent string). This means that events coming from the JavaScript SDK will automatically use Optimizely's bot filtering.
To filter traffic from bots in Full Stack experiments, you can use Optimizely's
JavaScript SDK or implement one of the following customizations:
- If you can identify bots on your own, use
getVariation()
for bots and activate()
for real visitors. This will forego dispatching a decision event for bots, which means Optimizely will not count them on the Results page.
- Use
getVariation()
instead of using activate()
in your backend. Include the Javascript SDK on the page and call activate()
on the page instead of in your backend. This moves the decision event to the front end, where user agents are captured and bot filtering will take effect.
- Customize the Optimizely SDKs to pass each visitor's user agent string in the User-Agent request header when dispatching events. This solution will probably require a non-trivial amount of work.
Since the
IAB/ABC bot list relies on the browser's user-agent strings to identify bots, Optimizely bot filtering has no effect on Full Stack results.
For more details on
implementing bot and spider filtering in Full Stack, see our developer documentation.
No. Optimizely Web automatically applies bot and spider filtering to all traffic as long as the
User-Agent header is present in the request.
Traffic from bots and spiders that are filtered is not included in the MUV allotment for your account.
We will waive any overage charges incurred as a result of bot or spider traffic. Please contact your Customer Success Manager if you believe you have incurred overage charges due to bot or spider traffic.
Contact your Customer Success Manager to learn more about your site’s historical bot and spider traffic levels.