AI crawlers and files
Should I block GPTBot and other AI crawlers?
Short answer
It depends on your goal. Blocking training crawlers such as GPTBot and ClaudeBot keeps your content out of future training but does not stop assistants that search the web. If you want to be cited, allow the search and user-initiated crawlers. Blocking everything makes your brand harder for assistants to learn about.
Two different goals
- Keep my content out of AI training. Block the training crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended.
- Be found and cited by AI assistants. Allow the search and user-initiated crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User and Perplexity-User.
Many sites choose a middle path: block training, allow search and user fetches. Publishers with paid content often block more. A business that wants customers to find it through assistants usually blocks less.
A middle-path robots.txt
# Allow AI search and user-initiated fetches
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Opt out of training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Things to keep in mind
- Blocking stops future collection by crawlers that honour robots.txt. It does not remove content that was already collected.
- Whether blocking training changes how well a future model knows your brand is uncertain. Weigh it against your reasons for blocking.
- Your firewall or CDN bot rules can block crawlers even when robots.txt allows them. Check both.
How SageRank helps
The free SageRank report shows which of ten AI crawlers your robots.txt blocks. Check yours with sagerank.io/yourdomain or paste the file into the AI crawler checker.