The AI Crawlers Split Into Three. Most robots.txt Files Still Name One.

The file still returns 200. It stopped being true in 2024.

On 20 August 2026 I ran a full reconciliation of sagentix.ca/robots.txt against what the AI providers currently document — all eleven of them, page by page. What came back was not a problem with the file. It was a structural change in how the providers work, and the file was the instrument that made it visible.

The file named 20 AI agents, every one Allow — 21 User-Agent blocks counting the wildcard. GPTBot, ClaudeBot, PerplexityBot, Googlebot: a deliberate, well-built file, and a better one than most sites carry. It was also a 2024 file, and it named anthropic-ai and Claude-Web — two strings that appear in none of Anthropic's current documentation — while naming neither Claude-User nor Claude-SearchBot, which did not exist when it was written.

Nothing was blocked. No monitor fired, the file returned 200, and it parsed cleanly. That is the whole finding: a correct file and a current file are different things, and no instrument on your side of the wire can tell them apart.

The spine: the providers split one crawler into three — one that trains, one that fetches when a user asks, one that indexes for search. A file written before that split names the training agent and misses the two that decide whether an assistant can reach and cite you. Nothing breaks when it happens, so nothing tells you.

What actually changed

The providers split one crawler into three, and they did it for a reason that makes sense once you see it: the three jobs have genuinely different consequences for a site owner, so they gave each one its own control.

Anthropic's documentation is the clearest statement of the pattern. It describes "the three robots that Anthropic uses" and separates them by function (Anthropic, 2026):

• ClaudeBot collects web content that may contribute to model training.
• Claude-User runs when a person asks Claude a question. Anthropic's own description of what disabling it does is the important sentence: it "prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search" (Anthropic, 2026).
• Claude-SearchBot indexes content for search-result quality.

Amazon documents the same three-way split — Amazonbot, Amzn-SearchBot, Amzn-User — and is explicit that they are separate controls: "Each user agent setting is independent of the others" (Amazon, 2026). Amzn-User "supports user actions, such as responding to Alexa queries that require up-to-date information" (Amazon, 2026).

Meta documents four crawlers rather than three, and the extra one is the agentic case. Meta-WebIndexer "navigates the web to improve Meta AI search result quality for users"; Meta-ExternalFetcher "fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities" (Meta, 2026).

OpenAI runs GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user-initiated fetches, plus OAI-AdsBot for ad landing pages (OpenAI, 2026) — but its answer on the middle one is not Anthropic's, and the difference is the most useful thing on the page. OpenAI states that "ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply" (OpenAI, 2026). For the search side it points owners elsewhere, recommending OAI-SearchBot as the agent to use in robots.txt for managing search opt-outs and automatic crawl (OpenAI, 2026). Anthropic presents Claude-User as an agent your file does govern, and warns that disabling it reduces your visibility. Same three jobs; two vendors, two different answers to whether your file governs the second one.

Perplexity runs PerplexityBot and Perplexity-User (Perplexity, 2026). DuckDuckGo added DuckAssistBot beside DuckDuckBot (DuckDuckGo, 2026). Apple runs Applebot and Applebot-Extended (Apple, 2026).

Three jobs. Train, fetch-for-a-user, index-for-search. Once you know the shape, you can read any vendor's page in thirty seconds and say which of the three you have named.

The misread

Most people who notice this will reach the same conclusion I first did: add the missing names to the allow-list. That is the misread, and it matters because it points you at the wrong file.

If your robots.txt is a permissive file — User-agent: * with Allow: / and a handful of private paths disallowed — then the agents you failed to name were never blocked. The wildcard already granted them exactly what the named blocks granted. Mine was one of these. The practical damage was zero. What I had was a maintenance failure wearing the costume of a technical one.

The file that actually costs you something is the one with a Disallow in it.

Consider a site that decided, reasonably, to keep its content out of AI training while staying visible in AI search. In 2024 that was one line per vendor: disallow ClaudeBot, disallow GPTBot, leave everything else alone. Today that same file blocks training and permits both retrieval agents — which is what its author wanted, by luck.

Now reverse it. A site that wanted to stop all AI traffic wrote Disallow: / for ClaudeBot and GPTBot and considered the job done. That file now blocks training and permits Claude-User, Claude-SearchBot, ChatGPT-User and OAI-SearchBot — the three-way split moved two of the three jobs outside the rule its author wrote. The intent and the instruction have come apart, silently, and the file still looks deliberate.

Both sites have a file that reads as considered. Only one of them still says what its author meant. Neither owner has any way to tell from the outside, because a robots.txt never reports what it failed to match.

Google is the reason you cannot do this by log inspection

The obvious workaround is to skip the documentation and read your access logs: whatever user-agents show up, name those. That fails on Google, and Google is not a small exception.

Google's crawler documentation states plainly that "Google-Extended doesn't have a separate HTTP request user agent string" (Google, 2026). It is a robots.txt control token and nothing else — crawling happens under existing Google user-agents, and the token governs whether the content is used for Gemini and Vertex AI grounding. It will never appear in a log file, because nothing ever sends it.

So the log-inspection method returns a list that is missing the single token controlling whether Google's AI products may use your content. A method that looks empirical, gives a confident answer, and cannot see the thing you were checking for is worse than no method — it retires the question.

The same trap sits one layer down, and it is set for careful readers specifically. Google-Extended does not appear on Google's crawlers overview page at all — it is documented on the special-case crawlers page. Read only the overview, as any reasonable person would, and the evidence says it has been retired: a fetch that succeeded, an answer that was confident, and nothing anywhere to signal the gap. This is precisely why the check has to be a named set of pages rather than a search.

What I found when I checked all of them

The interesting question was whether Anthropic was an outlier. It is not — the same shape appeared in four of the eleven vendors, and in each case the name carried forward was the training crawler:

Vendor	Documented as at 2026-08-24	My file named
Amazon	Amazonbot, Amzn-SearchBot, Amzn-User	Amazonbot
Meta	Meta-ExternalAgent, Meta-ExternalFetcher, Meta-WebIndexer, facebookexternalhit	Meta-ExternalAgent, plus FacebookBot
DuckDuckGo	DuckDuckBot, DuckAssistBot	DuckDuckBot
Google	Googlebot, Google-Extended, GoogleOther, Google-CloudVertexBot, Google-Agent, Google-NotebookLM	Googlebot, Google-Extended

In every one of the four, the name that carried forward was the training crawler and the ones absent were the two that decide whether an assistant can reach and cite the site. That is not four coincidences. It is one industry-wide restructuring, arriving quietly and at different times on eleven different documentation pages.

FacebookBot is worth its own sentence. It appears nowhere in Meta's current crawler documentation, which lists four names and does not include it. It does exactly what anthropic-ai and Claude-Web do in a 2024 file: occupy a line, match nothing, and make the file look current.

One name on the list has no vendor documentation at all. ByteDance publishes no crawler page I could reach — developer.bytedance.com returns 404 and bytespider.bytedance.com does not resolve — so Bytespider is attested only by community projects. It stays, with the exception written into the file beside it — an undocumented name you have decided to keep is a different thing from one nobody has looked at.

Why this decays and nothing tells you

Every failure mode here shares one property: the artifact keeps working while it stops being true.

A blocked crawler produces symptoms. A stale user-agent string produces nothing at all. It matches no requests, appears in no log, triggers no alert, and leaves a file that reads as carefully maintained. The only instrument that detects it is re-reading the vendor's documentation and comparing — and that is a task nobody schedules, because there is no event that prompts it.

The vendor documentation moves too. Of the eleven pages I checked, three had relocated: Google's crawler docs to a new path, Perplexity's to a different section, Anthropic's from support.anthropic.com to support.claude.com. All three still resolve through redirects, so a link check passes and a bookmark still works. The agent names had not changed in those three cases — but a process that treats "the link still works" as "the content is current" would not have known either way.

Two of the eleven pages do not render without JavaScript. Bing's crawler page returns a JavaScript shell whose extractable text is about ninety characters; Meta's returns HTTP 400 to a browser user-agent, reproducibly. Grep either result for a user-agent string and you get zero matches, which reads exactly like this vendor documents no crawlers. It is not a small distinction. A failed fetch and an empty subject produce identical output, and only one of them is a finding.

How Sagentix handles this

The check runs as a registry and a schedule rather than an intention. Every agent string, its role, the vendor page that documents it, and a canary token that proves a fetch actually worked — because a fetch that fails must never read as an absence. A weekly job re-reads all eleven pages and reports drift: agents no longer documented, new agent-shaped names on the page, documentation that moved, and any gap between what the vendors document and what the site's own file says. It is silent on a clean week and notifies on a change.

This is the same evidence discipline that runs on every client deliverable — 1,412 curated IP artifacts behind a 22-check quality gate, a 6–8 week engagement, CA$4K–$50K, with a Phase 1 money-back guarantee (subject to terms). The Phase 09 Digital Audit treats AI-retrieval configuration as a dated observation with a re-check interval, not a one-time fix, for exactly the reason this post exists Sagentix Phase 09 Digital Audit, 2026.

The general lesson is the one worth taking, and it is not about robots.txt. Expertise produces a correct artifact on the day it is written. Nothing about expertise keeps that artifact correct while the thing it describes moves underneath it. Only a re-read on a schedule does that — which is why the interesting deliverable here was never the corrected file. It was the weekly job that will catch the next change, and the one after.

Three things you can do with this

Read your own file against the three jobs, not the vendor names. Open yourdomain.com/robots.txt on your phone. For each vendor you have named, ask which of train / fetch-for-a-user / index-for-search you have addressed. If you named one string, you probably addressed one job. Twenty minutes, no tools, no subscription. If you have any Disallow line at all, do this today rather than this quarter — that is the file where intent and instruction come apart.

Put a re-read on the calendar, quarterly. The names change on the vendors' schedule, not yours, and nothing about the decay is observable from your side. A recurring reminder to re-read the eleven pages beats any amount of care taken once. Fetch each vendor's page with a real browser user-agent, and confirm each fetch returned actual content before concluding anything is missing.

Or have it audited as part of a broader diagnostic, which is what the Phase 09 Digital Audit does — with the standing caveat that the audit is only worth what its re-check interval is worth. If you take one idea from this piece and never speak to me, take the first one; it is where nearly all the value sits.

A last note on scope: Sagentix advises on go-to-market strategy. I do not deliver certification, audit or legal services, and nothing here is a route into selling any.

References

• Amazon. (2026). Amazonbot. Amazon Developer. https://developer.amazon.com/amazonbot
• Anthropic. (2026, April 7). Does Anthropic crawl data from the web, and how can site owners block the crawler? Anthropic Help Center. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
• Apple. (2026). About Applebot. Apple Support. https://support.apple.com/en-us/119829
• DuckDuckGo. (2026). DuckDuckBot. DuckDuckGo Help Pages. https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot/
• Google. (2026). Google crawlers and fetchers: Special-case crawlers. Google Search Central. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
• Meta. (2026). Web crawlers. Meta for Developers. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers
• OpenAI. (2026). OpenAI bots. OpenAI Developers. https://developers.openai.com/api/docs/bots
• Perplexity. (2026). Perplexity crawlers. Perplexity Docs. https://docs.perplexity.ai/docs/resources/perplexity-crawlers