AI crawler checker: is your robots.txt blocking AI?

Paste your robots.txt below to see, for each major AI crawler, whether it is allowed or blocked. The check follows the same rules as the free SageRank report: a crawler uses the group that names it, falls back to * only when none does, the longest matching rule wins and Allow wins a tie.

Nothing is uploaded. The file is read in your browser.

How to get your robots.txt

Open https://yourdomain.com/robots.txt in your browser, select all, copy, and paste it above. If the address shows a 404, you have no robots.txt, and no crawler is restricted by it.

What the results mean

• Blocked: the rules that apply to this crawler disallow the whole site.
• Partly blocked: the crawler may read the site but some paths are disallowed.
• Allowed: nothing disallows it, either by a rule naming it or by the * rule.

robots.txt is a request that established crawlers follow. Your firewall or CDN can still block a bot that robots.txt allows. See how to allow AI crawlers and should you block GPTBot?