Written by a person from the record · 2026-07-28 07:40:00 UTC

Fourteen sites served the same 501 bytes of robots.txt, and the bytes name who wrote them

A declared sample of 400 domains was asked for its robots.txt on 28 July 2026. 198 answered with one. Fourteen of those files contain 1,834 bytes that are identical across all fourteen, at the same offset — but only 501 of them sit inside the comment marking what the delivery network inserted. The marked part refuses eight AI crawlers. The 1,333 unmarked bytes before it are a legal notice, written in the site's own voice, asserting terms as a condition of access and reserving rights under EU copyright law. In none of the fourteen does the text outside the marked block name any of those crawlers, so nothing was overruled: the decision appears in a file the owner publishes and is absent from everything the owner wrote.

Correction, 28 July 2026 08:20 UTC

This finding was published at 07:40 UTC stating that 501 identical bytes appear in fourteen files. The figure was right and it was not the whole insertion. Checked afterwards: in all fourteen files the marked block begins at byte 1,334, and the 1,333 bytes before it are also identical across all fourteen. So 1,834 bytes are the same in every file, and the comment marking what was inserted encloses 501 of them — 27%.

The unmarked 1,333 bytes are not rules. They are a legal notice, and they are written in the first person of the website:

As a condition of accessing this website, you agree to abide by the following content signals…

ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

Between those two paragraphs it defines what search, ai-input, ai-train and use mean — the vocabulary of the Content-Signal line that appears inside the marked block.

What this changes: the earlier text counted those 1,333 bytes as the site's own, because the boundary comment says the insertion starts after them. It does not change any figure below. The preamble names no crawler, which is checked directly in the evidence chain, so the contradiction count is still zero.

What it adds is the sharper version of the same observation. A file that states terms of access and reserves rights under copyright law, in a site's own voice, sits outside the marker that identifies what the site did not write. Whether that boundary is meant to delimit rules rather than prose is not knowable from a fetched file; what is measurable is that a reader following the marker attributes those bytes to the site.

What was observed

Between 07:04:29 and 07:14:29 UTC on 28 July 2026, this site requested /robots.txt once from each of 400 domains. The population is a published ranking with a permanent identifier and the sample is a fixed stride through it, so the same 400 domains can be derived by anyone. The method, the rate and the terms were written down before the first request.

Fourteen of the 198 files — 7.1% of the files, 14.7% of the answers that came through that network — contain a section delimited by # BEGIN Cloudflare Managed content and its closing comment. All fourteen sections are identical byte for byte: 501 bytes, ten user-agent groups, in the same order, on sites that share nothing else in this sample.

Nine of the ten groups name a crawler and refuse it with Disallow: /. Eight of those nine are AI crawlers; the ninth is the network's own browser-rendering crawler. The tenth group is not a refusal at all — it applies to every client, allows the whole site, and carries a Content-Signal line stating a policy about training and reuse. That is a different kind of sentence: it asks to be honoured rather than blocking a request, and it is counted separately here rather than added to the eight.

What was not found

Not one of the fourteen files contradicts itself. A contradiction would be an inserted Disallow for a crawler that the site's own section names and allows, and in this sample there are none — because in all fourteen cases the site's own text never mentions those crawlers at all. There was nothing there to overrule.

This survey was built to count contradictions, having found one on this site: the origin file here welcomes those crawlers by name while the edge refused them. Naming an AI crawler in order to permit it turns out to be unusual. Across 198 files it did not happen once.

The weaker result is the published one. What the sample supports is narrow and worth stating on its own terms: fourteen sites publish a decision about eight AI crawlers that appears nowhere in anything they wrote.

What it means

A robots.txt is read as a statement by the site. It is the one file whose entire purpose is to say what its publisher permits, and every automated client that respects it treats it that way. In these fourteen cases part of that statement was composed elsewhere, and the file says so — the network signs its own insertion with a comment, which is the only reason this is measurable at all.

Byte-for-byte identity across fourteen unrelated sites is the part that needs no argument. Fourteen people writing their own rules do not produce the same 1,834 bytes in the same order, starting at the same offset.

Separately, 20 sites in the sample refuse an AI crawler in their own text with no insertion involved — more than the fourteen. Those are decisions somebody wrote down, and they are counted apart for exactly that reason.

Limits

• Nobody's intent was measured, and none is claimed. Whether these fourteen operators enabled this deliberately, inherited it as a default, or would recognise it if shown, is invisible from outside. A file that publishes a rule its owner did not type is evidence of that and of nothing further. The one case where the operator did not know is this site, which is a sample of one and is us.
• No domain is named. Reporting how many sites carry a rule they did not write is a measurement; reporting which ones is a list of other people's configurations published without an opportunity to object.
• An insertion that does not announce itself is invisible to this method. One network labels its additions, and that is the one this counts. A rewrite with no marker would read as the owner's own text here, and this page would have no way of knowing.
• Reachability was measured from one place. 128 domains did not answer, including twenty that reset the connection and also failed to resolve in DNS — filtering between this machine and them rather than those sites being down. A domain that did not answer is never counted as a domain without a robots.txt.
• Only a bare Disallow: / counts as a refusal. Narrower rules are real and are not a closed door; counting them would inflate every figure above.
• A wildcard is not a decision about a named crawler. A file saying User-agent: * has not decided anything about GPTBot, and is counted as not having mentioned it.
• 400 domains, one morning, one sample. 7.1% is a proportion of this sample and not an estimate for the web. A second run drawn the same way would land somewhere near it or would not, and that is a question for a second run.

Why this was worth recording

This site spent three days measuring what crawlers did to it, and then found that the file it used to invite them had been answering on its behalf. The obvious question is how far that extends, and it is answerable without asking anybody, because the intervention labels itself.

The answer is smaller than the question. That is the result, and it is published at the size it came in.

Evidence chain

This finding was not written from memory. Each figure below came from a query against the immutable request record, and each query is shown so it can be run again — by us, on this page, or by anyone reconstructing the same instrument.

What was measured

Each figure is stated first as what it means, which does not depend on how this system happens to store anything today. The query underneath it is implementation — it would be written differently on different storage, and it is shown so the figure can be checked, not because it is part of the claim.

SELECT COUNT(*) AS n FROM RobotsObservation WHERE surveyId = ?

SELECT COUNT(*) AS n FROM RobotsObservation WHERE surveyId = ? AND errorCode IS NULL

SELECT COUNT(*) AS n FROM RobotsObservation WHERE surveyId = ? AND errorCode IS NOT NULL

SELECT COUNT(*) AS n FROM RobotsObservation WHERE surveyId = ? AND httpStatus = 200
     AND lower(ltrim(body)) NOT LIKE '<!doctype%'
     AND lower(ltrim(body)) NOT LIKE '<html%'

SELECT COUNT(*) AS n FROM RobotsObservation WHERE surveyId = ? AND errorCode IS NULL
     AND headersJson LIKE '%"cf-ray"%'

SELECT COUNT(*) AS n FROM RobotsObservation WHERE surveyId = ? AND httpStatus = 200
     AND lower(ltrim(body)) NOT LIKE '<!doctype%'
     AND lower(ltrim(body)) NOT LIKE '<html%' AND body LIKE '%BEGIN Cloudflare Managed content%'

SELECT COUNT(DISTINCT substr(body, instr(body, '# BEGIN Cloudflare Managed content'), 501)) AS n
   FROM RobotsObservation WHERE surveyId = ? AND body LIKE '%BEGIN Cloudflare Managed content%'

SELECT COUNT(DISTINCT instr(body, '# BEGIN Cloudflare Managed content')) AS n
   FROM RobotsObservation WHERE surveyId = ? AND body LIKE '%BEGIN Cloudflare Managed content%'

SELECT COUNT(DISTINCT substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)) AS n
   FROM RobotsObservation WHERE surveyId = ? AND body LIKE '%BEGIN Cloudflare Managed content%'

SELECT MAX(instr(body, '# BEGIN Cloudflare Managed content') - 1) AS n FROM RobotsObservation WHERE surveyId = ? AND body LIKE '%BEGIN Cloudflare Managed content%'

SELECT COUNT(*) AS n FROM RobotsObservation WHERE surveyId = ? AND body LIKE '%BEGIN Cloudflare Managed content%' AND (
        substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)
      || substr(body, instr(body, '# BEGIN Cloudflare Managed content') + 501) LIKE '%GPTBot%'
     OR         substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)
      || substr(body, instr(body, '# BEGIN Cloudflare Managed content') + 501) LIKE '%ClaudeBot%'
     OR         substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)
      || substr(body, instr(body, '# BEGIN Cloudflare Managed content') + 501) LIKE '%CCBot%'
     OR         substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)
      || substr(body, instr(body, '# BEGIN Cloudflare Managed content') + 501) LIKE '%Google-Extended%'
     OR         substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)
      || substr(body, instr(body, '# BEGIN Cloudflare Managed content') + 501) LIKE '%Applebot-Extended%'
     OR         substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)
      || substr(body, instr(body, '# BEGIN Cloudflare Managed content') + 501) LIKE '%Bytespider%'
     OR         substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)
      || substr(body, instr(body, '# BEGIN Cloudflare Managed content') + 501) LIKE '%meta-externalagent%'
     OR         substr(body, 1, instr(body, '# BEGIN Cloudflare Managed content') - 1)
      || substr(body, instr(body, '# BEGIN Cloudflare Managed content') + 501) LIKE '%Amazonbot%'
   )

Every figure still returns what it returned at publication.

Governed by

• I. Reality is not interpretation — Reality ≠ Interpretation
• IV. Replay guesses nothing — Every figure must be reproducible from the record
• VI. The record is honest about its size — Sample size is stated; absence is reported as absence

These are the standards this finding was held to, not a claim that it is beyond question. The constitution states them in full.

Other findings

• One address requested 165 distinct paths within 17 seconds
• One address requested 31 distinct paths within 6 seconds
• One address requested 47 distinct paths within 5 seconds
• Which content formats are actually fetched
• One address requested 22 distinct paths within 3 seconds

All findings · Live record · Methodology