AI crawlers: which bots visit your site, and what they do

An AI crawler is a bot run by an AI company that fetches your pages. They aren't all after the same thing. Some gather content that may be used for training, some build the index behind an AI search feature, and some fetch one page because a person just asked about it. You can allow or block each kind separately.

TL;DR

An AI crawler is a bot run by an AI company that fetches your pages.
Some gather content that may be used for training, some build the index behind an AI search feature, and some fetch one page because a person just asked about it.
You can allow or block each kind separately.

An AI company's botFetches your pageAllow it or block it

Stickybeak's bot report sorts AI assistants, search engines and scrapers by what they are after, and recognises over 80 bots by name. It is in early access.

Get early access

The short version

Training crawlers (GPTBot, ClaudeBot) collect content that may feed model training. Blocking them is a choice about training.
Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the index an AI search feature draws on. Block these and your pages may not appear in those answers.
User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) visit a page because someone asked a question. Vendors differ on whether robots.txt applies to them.
Google-Extended is a robots.txt token, not a separate bot. Google says it doesn't affect Search inclusion or ranking.

Which to allow depends on what you want. If you want to be named in AI answers, leave the search crawlers and user-triggered fetchers alone. If you don't want your writing used for training, block the training crawlers. Each one follows its own rules, so they have to be handled one by one.

AI training

Collect content that may feed model training.

GPTBotOpenAI
ClaudeBotAnthropic

If you block themA choice about training. AI search and answers carry on.

AI search

Build the index an AI search feature draws on.

OAI-SearchBotOpenAI
Claude-SearchBotAnthropic
PerplexityBotPerplexity

If you block themYour pages may not appear in those answers.

AI answers

Fetch one page because a person just asked about it.

ChatGPT-UserOpenAI
Claude-UserAnthropic
Perplexity-UserPerplexity

If you block themVendors differ on whether robots.txt applies.

Three kinds of AI crawler, and the companies' own names for them. Each kind follows its own rules, so allowing or blocking one says nothing about the others.

The main AI crawlers

Everything in this table comes from each company's own documentation, checked on 8 October 2026 (links at the foot of the page). Vendors change these pages, so check the source before you rely on a detail.

Crawler	Run by	What the vendor says it's for	robots.txt
`GPTBot`	OpenAI	Used to make its generative AI foundation models more useful and safe (training).	"Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."
`OAI-SearchBot`	OpenAI	Used to surface websites in search results in ChatGPT's search features.	Sites opted out "will not be shown in ChatGPT search answers." Changes can take about 24 hours to apply.
`ChatGPT-User`	OpenAI	Certain user actions in ChatGPT and Custom GPTs.	"Because these actions are initiated by a user, robots.txt rules may not apply."
`ClaudeBot`	Anthropic	Collects web content that could potentially contribute to training its models.	Respects "do not crawl" signals. Anthropic supports a non-standard `Crawl-delay`.
`Claude-SearchBot`	Anthropic	Navigates the web to improve search result quality for users.	Honours robots.txt. Disabling it prevents indexing your content for search optimisation.
`Claude-User`	Anthropic	Accesses websites when a person asks Claude a question.	Respects robots.txt. Disabling it stops Claude retrieving your content in response to a user query.
`PerplexityBot`	Perplexity	Surfaces and links websites in Perplexity search results. Perplexity says it is not used to crawl content for AI foundation models.	Respects robots.txt.
`Perplexity-User`	Perplexity	Supports user actions: when a user asks a question it might visit a page to answer it.	Perplexity says it "generally ignores robots.txt rules" because the request is user-initiated.
`Google-Extended`	Google	A standalone token that manages whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.	Not a separate user agent. Google crawls with its existing user agents and uses the token as a control. It "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
`Applebot-Extended`	Apple	Lets publishers opt out of content being used to train Apple's general purpose foundation models.	Apple says it does not crawl webpages, and pages that disallow it can still appear in search results.

Two things in that table catch people out. First, blocking a training crawler does not remove you from the same company's search or answer features, because they are separate crawlers with separate rules. Second, vendors treat user-triggered fetchers differently: Anthropic says Claude-User respects robots.txt, while OpenAI and Perplexity say theirs may not.

Allowing or blocking them in robots.txt

Rules go in the robots.txt file at the root of your site. Each group starts with a User-agent line naming the crawler. Disallow: / asks it to stay out of the whole site, and an empty Disallow: allows everything.

To block the training crawlers but stay open to AI search and user requests:

User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: /

Anything not named keeps following your default rules, so OAI-SearchBot, PerplexityBot and the rest can still read the site. To block one section only, replace / with a path such as /members/.

Asked to stay out Disallow: /

GPTBotTraining, OpenAI
ClaudeBotTraining, Anthropic
Google-ExtendedA token, not a bot
Applebot-ExtendedA token, not a bot

Still reading the site Not named

OAI-SearchBotChatGPT search
Claude-SearchBotClaude's search
PerplexityBotPerplexity search
The user fetchersChatGPT-User, Claude-User, Perplexity-User
GooglebotGoogle Search is unaffected

Blocking the training crawlers leaves AI search, user requests and Google Search exactly as they were. The two tokens never visit; they are rules that Google's and Apple's existing crawlers read.

Before you copy that, two cautions:

Google-Extended covers more than training. Google describes it as controlling training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI, so blocking it is not purely a training decision.
robots.txt is a request. It tells well-behaved crawlers what you'd like. It doesn't enforce anything, and your CDN or host may have its own bot rules on top, so check those too.

To see what your own file does to a given crawler, paste it into the robots.txt checker, type the bot's name and test a URL.

Changes take time to be picked up. OpenAI quotes about 24 hours and Perplexity says up to 24 hours, so don't expect a new rule to apply on the next request.

How to see which AI crawlers visit you

AI crawlers are one kind of bot traffic, alongside search crawlers, monitors and scrapers. Most of these bots never run JavaScript, so a normal analytics script can't see them. Two places will:

Server or CDN logs. Search the user agent field for the crawler names in the table above, other than Google-Extended and Applebot-Extended. Those two are robots.txt controls rather than crawlers, so they never appear in logs: the fetches behind them show up under Google's existing user agents and Applebot, and logs can't show whether the content was later used for training. Each vendor lists the exact string it uses on the pages linked at the foot of this page.
A tool that reads those logs for you. The point is to see bots separately, so they don't inflate your visitor numbers.

A user agent is only a label that anyone can type, so a name in your logs is not proof of who sent the request. Anthropic publishes a list of its crawler IP addresses you can check against.

Request fromServer or CDN logAnalytics script

A personIn a browser

Log: SeenEvery request

Script: SeenTheir browser runs the script

GPTBotA crawler

Log: SeenUnder its user agent

Script: Usually notIt doesn't run JavaScript

Google-ExtendedA robots.txt token

Log: NeverLogged as Google's other crawlers

Script: NeverNot a visitor at all

Most AI crawlers skip JavaScript, so they're only in your server or CDN logs. And a name in a log is only a label: check it against the vendor's published IP addresses where they offer them.

How Stickybeak shows crawlers

Stickybeak is live analytics for every site you own, and it's in early access. Its bot report sorts AI assistants, search engines and scrapers by what they are after, using Cloudflare's data so crawlers that never run JavaScript are seen. It recognises over 80 bots by name and keeps them out of your visitor counts.

Stickybeak's bot report from the demo on our home page: Googlebot and Bingbot as search engines, GPTBot and ClaudeBot as AI training, ChatGPT and Perplexity as AI answering a question, each with a request count. — The bot report, from the demo on our home page. The sites and numbers are made up.

Stickybeak doesn't have a public price yet. If you'd like to try it when there's a spot, leave your email.

Get early access

Questions

What is an AI crawler?

A bot run by an AI company that fetches web pages. Some collect content that may be used to train models, some build the index behind an AI search feature, and some fetch a page when a person asks a question.

Does blocking GPTBot stop ChatGPT showing my site?

Not by itself. OpenAI says disallowing GPTBot means content should not be used to train its generative AI foundation models. Appearing in ChatGPT search is controlled separately by OAI-SearchBot.

Does blocking Google-Extended affect my Google rankings?

Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal. It controls whether content may be used for training Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

Do all AI crawlers obey robots.txt?

The vendors in this guide say their crawlers respect robots.txt, with exceptions for fetches a user triggers. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt.

How can I see which AI crawlers visit my site?

Crawlers don't run JavaScript analytics, so a normal tracking script misses them. Check your server or CDN logs for the user agent names, or use a tool that reads them for you. Stickybeak's bot report pulls from Cloudflare and sorts bots by what they are after.

Sources, checked 8 October 2026: OpenAI, Overview of OpenAI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User). Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? (ClaudeBot, Claude-User, Claude-SearchBot; page dated 7 April 2026). Perplexity, Perplexity crawlers (PerplexityBot, Perplexity-User). Google Search Central, Google's common crawlers (Google-Extended; last updated 14 July 2026). Apple, About Applebot (Applebot-Extended).