AI crawlers and files
Which AI crawlers exist, and what are their user agents?
Short answer
The main AI crawlers are GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-User from Anthropic, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Bytespider. Some collect training data, some build search indexes, and some only fetch a page when a person asks.
The list
| Name | Operator | Mostly used for |
|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used to train models |
| OAI-SearchBot | OpenAI | Showing sites in ChatGPT search results |
| ChatGPT-User | OpenAI | Fetching a page when a person asks ChatGPT to visit it |
| ClaudeBot | Anthropic | Collecting content that may be used for training |
| Claude-User | Anthropic | Fetching a page on a person’s request |
| anthropic-ai | Anthropic | An older token; ClaudeBot is the current one |
| PerplexityBot | Perplexity | Building the index behind Perplexity search |
| Perplexity-User | Perplexity | Fetching a page on a person’s request |
| Google-Extended | A robots.txt token that controls use of your content for Gemini training and grounding. It is not a separate crawler and does not affect Search | |
| Applebot-Extended | Apple | A token that controls use of Applebot-crawled content for Apple’s AI models |
| CCBot | Common Crawl | A public web archive that many AI training sets draw on |
| Bytespider | ByteDance | ByteDance’s crawler |
Read this before you act on the list
- These lists change. Confirm the current names in each operator’s own documentation before editing robots.txt.
- robots.txt is a request, not a lock. Established crawlers follow it; others may not.
- A user-initiated fetch (ChatGPT-User, Claude-User, Perplexity-User) is a person asking an assistant to read your page. Blocking it means the assistant cannot read your page for that person.
How SageRank helps
The free SageRank report reads your robots.txt and tells you whether each of ten AI crawlers is blocked: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider and Applebot-Extended. You can also paste a robots.txt into the free AI crawler checker.