Research
How AI-ready are the most visited domains on the web?
Summary
We ran the checks behind the free SageRank report on the 1,000 most visited domains on the Tranco list. We could read the home page of 539; many of the rest are infrastructure domains or block automated visitors. Of those 539, 22.2% block at least one of ten AI crawlers, 25.1% publish an llms.txt and 74.6% serve readable text without JavaScript.
What we found
- 22.2% of the readable sites block at least one of ten AI crawlers site-wide in robots.txt. 16% block GPTBot.
- 25.1% publish an
llms.txt, and 5.6% also publishllms-full.txt. - 47.1% carry schema.org structured data in the page HTML, 43% describe the organisation, and 14.4% mark up content such as articles or products.
- 74.6% put at least 150 words of readable text in the HTML a simple crawler receives, without running JavaScript.
Which AI crawlers are blocked
Share of the readable sites whose robots.txt disallows the whole site for each crawler (531 robots.txt files read). A site that blocks only some folders is not counted as blocking.
| Crawler | Operator | Blocked |
|---|---|---|
GPTBot | OpenAI (training) | 16% |
ChatGPT-User | OpenAI (live browsing) | 10.9% |
OAI-SearchBot | OpenAI (search) | 9.2% |
ClaudeBot | Anthropic | 16.2% |
anthropic-ai | Anthropic (legacy) | 11.3% |
PerplexityBot | Perplexity | 12.4% |
Google-Extended | Google Gemini | 14.9% |
CCBot | Common Crawl | 18.6% |
Bytespider | ByteDance | 18.8% |
Applebot-Extended | Apple Intelligence | 12.6% |
Page basics
Share of the readable home pages that pass each check in the free report (base: pages where the check could run).
| Check | Pass |
|---|---|
| A page title | 72.7% |
| A meta description | 62.6% |
| A canonical link | 69.5% |
| Open Graph sharing tags | 56.6% |
| A sitemap | 69.9% |
| HTTPS with http redirecting to it | 86.4% |
How we did it
- Who: the first 1,000 domains on Tranco list
PYK6J, a public ranking built from several traffic sources. Many are infrastructure, advertising or API hosts with no home page meant for people, and the rest are very large sites, so the results say little about small businesses. - What: for each domain we fetched the home page,
/robots.txt,/llms.txt,/llms-full.txtand the sitemap once, with SageRank's own free-report checks, through a fetcher that only reaches public hosts and respects size and time limits. - Counted: 539 of 1000 home pages (53.9%) could be read. Domains with no readable home page, sites that block automated visitors and sites that time out are left out, and a check that could not run is left out of its own percentage.
- Simple crawler view: pages are read as a crawler that does not run JavaScript sees them. A site that builds its content in the browser can score lower here than a person would judge it.
- Blocked means: robots.txt disallows the whole site for that crawler, directly or through the
*rule. It does not mean the site actually refuses the bot at its firewall. - Limits: home pages only, one visit on 30 October 2026, results change over time. We name no sites. To cite or discuss the data, write to [email protected]. The totals are in aggregate.json.
What to do about it
- Decide on purpose which AI crawlers you want. See should you block GPTBot? and test your file with the AI crawler checker.
- Add an llms.txt if you want one. It takes under an hour.
- Make sure your key content is readable without JavaScript and add structured data.