The robots.txt survey

This site reads other sites' robots.txt to count how often a content delivery network has written something into it that the owner did not write.

If you found this in your logs

A request from AgentShieldObservatory/0.1 (+https://agentshieldaidefense.com/survey) fetched /robots.txt from your domain and nothing else. One request, no other path, no cookies, no scripts, no second visit within a run. The file was stored as it was served and is used only to count — no domain is ever named in anything published here, including yours.

To be excluded, refuse the agent in the file itself:

User-agent: AgentShieldObservatory
Disallow: /

That is read before the next run and honoured. There is an obvious circularity in asking a survey about robots.txt compliance to prove its own by obeying robots.txt, and the only answer to it is that the fetches are recorded here and can be checked against what this page claims.

Why

On 27 July 2026 this site discovered that its own robots.txt was being served with nine crawler groups and Disallow: / prepended to it by the CDN in front of it. The file at the origin welcomed those crawlers by name. For an unknown period, the site had been telling the crawlers it exists to observe not to come, in a voice that was not its own, and nothing at the origin could see it.

The obvious next question is how many other sites are in that position. It is answerable because the intervention signs itself: the CDN writes a boundary comment around what it adds, so separating the inserted part from the owner's part needs no guess about what the origin "probably" served.

What the record shows

Of 400 domains, 272 answered and 128 did not. 198 of the answers were a robots.txt; 74 were something else, usually an HTML error page returned with status 200.

95 of the 272 answers came through Cloudflare, by its own response header. 14 of the 198 files (7.1%) carried a block that Cloudflare had inserted, marked by the boundary comment it writes around its own additions. 14 of those close the site to at least one of the 15 AI crawlers this survey asks about.

All 14 of those blocks are byte for byte identical — 10 user-agent groups in 501 bytes, the same order every time, on sites that have nothing else in common. Whatever else is true of them, they were not written by 14 different people.

9 of those groups name a crawler and refuse it — 8 of them AI crawlers this survey asks about, and the rest the CDN's own. The remaining group applies to everyone and carries no Disallow at all: it states a Content-Signal policy about training and reuse. That is a different kind of sentence from a refusal — it asks to be honoured rather than blocking a fetch — and it is counted apart from the refusals rather than added to them.

None of them contradicts the owner's own file. A contradiction would be an inserted Disallow for a crawler that the owner's own section names and allows, and in this sample there are none — because in all 14 cases the owner's file never mentions those crawlers at all.

That is a weaker result than the one this survey was built to look for, and it is reported as the weaker one. This site found the contradiction on itself, where the origin file welcomed those crawlers by name; naming them is unusual, and a sample of 198 files contains no other example. What the sample does show is 14 sites serving a decision about 8 AI crawlers that appears nowhere in anything they wrote. Whether they made that decision somewhere else, or would recognise it, is not visible from outside and is not claimed here.

Separately, 20 sites close the door to an AI crawler in their own text, with no inserted block involved — more than the number carrying an injection. Those are decisions somebody wrote down, and they are counted apart from the injections for exactly that reason.

Which crawlers, and what the owner said about the same name

The third column is the contradiction. The fifth is not: a site that never mentioned a crawler has not been overruled, it has been answered for. Both are worth counting and they are different things.

Domains that did not answer

These are the runtime's own error codes, kept verbatim rather than sorted into categories. Some of them are not about the domain at all — see the limits below.

Limits

Reachability was measured from one place. Of the first eight domains ever fetched, two answered with a connection reset and failed to resolve in DNS — the signature of network filtering between here and them, not of those sites being down. Nothing in the instrument can tell the two apart, so the vantage point is recorded with every run and a domain that did not answer is never counted as a domain without a robots.txt.

A wildcard is not an answer about a named crawler. A site whose file says User-agent: * has not decided anything about GPTBot, so it is counted as not having mentioned it. Reading a wildcard as a decision would report sites as having closed a door they never considered.

Only a bare Disallow: / counts as closed. Narrower rules are real and are not a closed door. Counting them would inflate every figure on this page.

An intervention that does not announce itself is invisible here. This survey finds one CDN's managed block because that CDN labels it. A rewrite with no marker would be counted as the owner's own text, and this page would have no way of knowing.

Nobody's intent is measured. A file that contradicts itself is evidence of a contradiction. Whether the owner knows, chose it, or would change it is outside what a fetched file can show, and the numbers above say nothing about it.

Population tranco-46ZYX · sample drawn 2026-07-28 · analysis sa-1 · last run completed 2026-07-28 07:14:29 UTC