AI crawlers and files

How do I allow AI crawlers in robots.txt?

Reviewed 2 November 2026 by the SageRank team

Short answer

To allow AI crawlers, make sure no rule disallows them. A robots.txt with no restrictions, or with User-agent: * followed by Allow: /, lets them in. Name a crawler only to give it different rules. Then check that your firewall or CDN is not blocking the same bots.

Allow everything

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Allow the AI crawlers by name

You do not need to name them, but doing so makes your intent clear and protects you if a broader rule is added later.

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Mistakes that block crawlers by accident

Disallow: / under User-agent: *. A staging rule that reached production is the usual cause.
A broad rule such as Disallow: /*? or Disallow: /blog/ that hides the pages you most want read.
A bot-protection rule at your CDN or firewall that challenges unknown bots. robots.txt may say allow while the server says no.
A robots.txt that returns an error. A 5xx response can make crawlers treat the whole site as off limits for a while.

Test it

Fetch /robots.txt from the live domain, then read it for each crawler you care about. Paste it into the AI crawler checker to see the result for each one.

How SageRank helps

The free SageRank report scores any domain for SEO, AEO, GEO and security, adds one overall score, and lists what to fix first. Type sagerank.io/ in front of any address to run it.