AI crawlers and files

Is robots.txt still used, and what does it do?

Reviewed 2 November 2026 by the SageRank team

Short answer

Yes, robots.txt is still used. It is a public text file at the root of your site that tells crawlers which paths they may fetch, and it was formalised as a standard in 2022. Search engines and most AI crawlers read it. It controls crawling, not indexing, and it is a request, not a security measure.

What it controls

Crawling. Whether a crawler may fetch a URL.
Which crawlers. You can give each named crawler its own rules.
Where your sitemap is. A Sitemap: line points crawlers to it.

What it does not control

Indexing. A page blocked in robots.txt can still appear in results if other sites link to it. To keep a page out of the index, allow crawling and use a noindex robots meta tag or an X-Robots-Tag header.
Security. The file is public and obeying it is voluntary. Never use it to hide private pages. Put those behind sign-in.

A minimal file

User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Why it matters more now

Robots.txt is also how you tell AI crawlers what they may read. A rule meant for one bot can block them all by accident. See how to allow AI crawlers and test yours with the AI crawler checker.

How SageRank helps

The AI crawler access check in the free SageRank report reads your robots.txt and shows which of ten AI crawlers it blocks. Run it with sagerank.io/yourdomain.