AI crawlers and files
What is llms.txt and do I need one?
llms.txt is a proposed plain Markdown file at the root of a website that gives AI tools a short, curated guide to the site: what it is and which pages matter. It is a community proposal, not an official standard, and AI providers have not confirmed that they rank or cite sites because of it. It is cheap to add.
Where it comes from
The idea was proposed in 2024 as a way to help language models read a site at the moment someone asks a question. A page at /llms.txt holds a title, a one-line summary and links to the key pages, each with a short note. A longer /llms-full.txt can hold the full text.
What it can and cannot do
- It can give tools and agents that choose to read it a clean, curated map of your site, which is especially useful for documentation.
- It cannot force any assistant to name or rank you. It is not an official standard, and no major AI provider has said that it uses llms.txt as a ranking signal.
- It is not robots.txt. It does not allow or block anything. Use robots.txt for access rules.
Do you need one?
It takes under an hour, costs nothing and does no harm, so many sites add one. Keep your expectations modest and do not let it replace the basics: crawlable pages, clear facts and structured data.
How SageRank helps
The llms.txt check in the free SageRank report looks for /llms.txt and /llms-full.txt on your domain. Build one with the free llms.txt generator.