Filter bots and spiders

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Performance Edge
  • Optimizely Feature Experimentation
  • Optimizely Full Stack (Legacy)
  • Optimizely Personalization

Optimizely Performance Edge is a lightweight experimentation product that delivers significantly faster performance than previous versions of Optimizely Web Experimentation. It does this by relying on a streamlined "microsnippet" which limits the range of available features.

Internet bots and spiders are software applications that run repetitive, automated tasks over the Internet. Optimizely uses the IAB/ABC international spiders and bots list to classify non-human traffic and filter it out of your results. The IAB/ABC list is an actively maintained record of user agents used by known spiders and bots. The list is updated on a monthly basis.

How does bot and spider filtering affect my results?

Bots and spiders do not behave like real users. They typically run repetitive, automated tasks over your website or app at a much higher rate than real users, with the purpose of crawling your content and analyzing it. As a result, bots and spiders usually inflate the volume of traffic to your site and may skew your conversion metrics.

Removing bot and spider traffic from your results generally decreases the volume of traffic, such as the total number of visitors or sessions, and makes your measurements more accurate.

How does bot and spider filtering work?

Optimizely reads the User-Agent header for each request and determines whether it matches with an entry in the IAB/ABC list. If there is a match, Optimizely classifies the request as non-human and automatically excludes it from the results.

Are there any prerequisites for using bot and spider filtering?

Bot and spider filtering relies on the client’s User-Agent header to work. For web traffic, the User-Agent header is normally sent with every request, so Optimizely Web Experimentation and Optimizely Personalization customers automatically benefit from it.

Is bot and spider filtering available in Optimizely Feature Experimentation?

Yes. This capability is available for all Optimizely Feature Experimentation SDKs. You must first enable bot filtering in the Optimizely Feature Experimentation application and then implement the $opt_user_agent attribute unless you are using the JavaScript SDK in the browser. See Filter out bots.

Can I opt out of bot and spider filtering?

  • Optimizely Web Experimentation, Optimizely Performance Edge, and Optimizely Personalization – No, bot and spider filtering is automatically applied to all traffic as long as the User-Agent header is present in the request.
  • Optimizely Feature Experimentation – Yes, see Filter out bots for information on how to toggle bot filtering on and off. 

Does bot and spider filtering affect the way search engines crawl my site?

No, search engine crawlers will still see your site. To make sure that your SEO is not affected, see A/B testing and search engine optimization.