Stickybeak Docs Dashboard Get early access

Privacy and data

Bot filtering

Bot traffic is labelled and kept out of every number: visitors, pageviews, sessions, goals and revenue attribution. Bot filtering needs no setup, and a separate panel shows what visited.

How bots are spotted

Each visit is checked as it arrives, in roughly this order. The first match labels it.

  1. Signed agents. AI agents that sign their requests (they send a Signature-Agent header) are labelled with the agent's host.
  2. No user agent at all.
  3. Known bots by name. Over 70 rules recognise named crawlers and tools: search engines, AI crawlers and assistants, social preview fetchers, SEO tools, uptime monitors, Lighthouse, and HTTP libraries like curl and python-requests.
  4. Cloudflare's verified bots, when Cloudflare identifies one.
  5. Anything that calls itself a bot: user agents containing words like "bot", "crawl", "spider" or "headless".
  6. Automated browsers. The browser reports it's being driven by automation software, or identifies as headless Chrome.
  7. Non-browser clients whose user agent doesn't look like any browser.
  8. Data-centre networks. Visits from cloud hosts such as AWS, Google Cloud, Azure, DigitalOcean and Hetzner, where real people rarely browse from.
  9. Missing browser headers. Every mainstream browser sends its preferred languages; a request without them is treated as automated.

Once a session is labelled a bot, everything in it stays a bot, including later pages and goals.

AI crawler tracking

  • On the wall, a robot pill on a site's card shows how many bots visited in the last 30 minutes. Hover it for their names. The top bar shows the total filtered across all sites.
  • On a site page, Bots filtered sits with the headline numbers, and the Bots (not counted) panel lists each bot with its category.

Categories are: AI answers (fetched because a person asked an assistant), search indexes, AI training crawlers, other AI bots, SEO tools, social previews, monitoring, automation and headless browsers, scripts and HTTP clients, and other bots.

Crawlers that don't run JavaScript

Most crawlers never run scripts, so no analytics script can see them. For workspaces whose sites are on the Cloudflare account Stickybeak runs on, the site page adds a crawler panel built from Cloudflare's request analytics: which bots fetched your pages, how often, and the pages each fetched most. It updates hourly. Other workspaces don't see this panel.

Limits

  • A bot that drives a real browser from a home internet connection, with ordinary headers, can look like a person. No analytics tool catches all of those.
  • Some real people browse from data-centre networks, for example through a VPN or a cloud desktop. They're counted as bots.
  • You can't mark a bot as human, or add your own rules.

In privacy mode, bots are spotted the same way, but the raw user agent and network name aren't stored afterwards.