I built a site to watch AI crawlers. My CDN was quietly turning them away.

Five things sat between what my server sent and what anybody received. I found them by accident, and only because a number looked wrong.

Before you read any further, run this against your own domain. It takes ten seconds, and you may want the answer before you finish this sentence:

curl -s https://your-domain/robots.txt | grep -c "Managed content"

Anything other than zero means something is writing rules for your site that you did not write.

This is that site. One domain, thirty-nine pages, a database that records every request that arrives — the address, the user agent, every header, verbatim. The question is narrow on purpose: how do AI systems actually read the web? Not what the vendors say they do. What arrives at the door.

Everything below is a claim about bytes, and you should be able to fetch them yourself. The live figures update as requests arrive. The findings each carry the query that produced them. If I have this wrong, the material to prove it is public.

The site publishes what it finds, and it publishes its own rules. One of those rules says:

Every client receives identical bytes for the same URL.

Another says the crawlers are welcome, because a site built to observe them and then blocking them would be pointless.

Both of those statements were false. Not because my server was wrong — it sent exactly what I told it to. They were false because a claim about bytes is a claim about what a reader receives, and the origin server is not the last thing to touch them.

The thread I pulled

I had just added ETag headers so I could measure whether AI crawlers make conditional requests — whether they ask "has this changed?" before downloading a page again. It is a politeness question and a bandwidth question, and nobody publishes real numbers on it.

The header worked locally. It never reached the internet.

I assumed compression. It was not compression. I assumed a caching setting. It was not that either. I predicted the fix three times and was wrong three times, which is usually the point where you stop theorising and start measuring.

So I fetched a page from my own server, fetched the same page through the CDN, and compared the bytes.

origin: 11,670 bytes
edge:   11,958 bytes

Two hundred and eighty-eight bytes I did not write.

What was in them

<a href="https://agentshieldaidefense.com/cdn-cgi/content?id=…"
   aria-hidden="true" rel="nofollow noopener"
   style="display: none !important; visibility: hidden !important"></a>

A hidden link, injected immediately after <body>, with an id that changed on every single request. Invisible to a human. Perfectly visible to anything parsing HTML.

I followed it. It returns a machine-generated essay. On one fetch the title was The Labyrinth of Knowledge: An Odyssey into Epistemology. On the next fetch of the same URL: Atomic Physics: Unveiling the Mysteries of the Microcosm. Marked noindex, nofollow. No outbound links. Generated at the edge — my server has never recorded a single request for that path, and the path returns 200.

It is a decoy. Following it is a signal, and the signal it produces is this client is a bot.

Then I checked who got it.

Client	Received the injection
Desktop browser	yes
Googlebot	yes
curl	yes
No user agent at all	yes
python-requests	yes
Unrecognised bot	yes
GPTBot	no
ClaudeBot	no

The content varied by user agent. On a site whose stated rule is that content does not vary by user agent — and, worse, on the exact instrument built to measure whether other people serve different content to crawlers.

The population this site exists to observe most carefully is the one that does not declare a recognised identity. Those were precisely the clients being handed a hidden trapdoor.

Then I looked at robots.txt

This is the one that matters.

origin: 1,468 bytes
edge:   3,304 bytes

My file, the one in my repository, opens like this:

# AI crawlers and agents are welcome here.
# This site measures how AI systems read the web; blocking them would defeat it.

The file that actually left the edge opened differently. Above my text, prepended, was a block I did not write:

# BEGIN Cloudflare Managed content

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

# END Cloudflare Managed Content

Nine user-agent groups, every one of them told to leave. Directly above my own paragraph inviting them in.

One document. Two voices. Opposite instructions. One of the voices was not mine.

So how much crawler traffic did I actually get?

Sixty external requests arrived claiming one of the AI crawler identities this site checks for. That number looks like data until you ask where it came from.

	Requests
Claimed an AI crawler identity	60
…of which came from one single address	49 (81.7%)
…from the other eight addresses	11

One machine, presenting six different companies' crawler identities, produced four fifths of my AI crawler traffic. It was not crawling. It spent six seconds hunting for .env, .git/config and .aws/credentials, found nothing, and left — and it wore those identities because a great deal of infrastructure waves GPTBot through a filter it would stop anything else at.

Which leaves eleven requests, across eight addresses, in two days.

Every one of those claims is checked against the address ranges each vendor publishes for its own crawler. Across all sixty:

7 corroborated — the request came from inside the vendor's published range
30 contradicted — the vendor publishes a list and this address is not on it
23 uncheckable — no machine-readable list exists to check against

That last row is not an accusation of anybody. Anthropic and Common Crawl publish nothing you can verify against, so every one of their agents lands there however genuine it is. A vendor's silence must never be rendered as a client's guilt.

And the percentages are close to meaningless on their own, because one incident supplied four fifths of the sample. I am saying that in the same breath as the numbers, because a figure that hides its own dominant contributor is worse than no figure. The live version recomputes all of it, including what share the largest single address currently holds.

So: eleven requests that were not the impostor, over two days, from the crawlers of the largest AI companies in the world, at a site expressly built to be read by them.

I had been reading that as crawler behaviour. It may be nothing of the kind. It may be a measurement of an instruction I never issued.

I cannot tell you which, and that is the honest answer. I can tell you the block is gone now, and that the same number over the coming weeks will settle it.

Stop here for a second, because this part is not about me

I noticed this because measuring what reaches clients is the entire purpose of this site. I had a database of every request, a habit of comparing origin bytes against served bytes, and three days of paranoia about self-reported evidence.

You almost certainly have none of that, and you have the same CDN or one very like it.

If your business depends on being found — by a search engine, by an AI assistant, by anyone who reads pages instead of looking at them — then the file that tells those systems what they may read is the single most consequential text you publish. It is also the one nobody ever opens again after the day they write it.

Mine was rewritten. I found out on day three, by accident, while investigating something else entirely.

And the ETag, finally

Once the injection stopped, the bytes matched — origin and edge, identical, every page. The ETag still did not come through.

Content type	gzip	ETag survives
`text/html` (incl. 404s, incl. `HEAD`)	yes	no
`application/rss+xml`	yes	yes
`application/json`	yes	yes
`application/xml`	yes	yes
`text/plain`	yes	yes
`text/markdown`	no	yes

Compression is ruled out — compressed non-HTML keeps its validator, and uncompressed markdown keeps its validator too. The injection is ruled out; it is gone and the bytes are identical. Status is ruled out; a 404 loses it too. Method is ruled out; HEAD loses it too.

The only variable left is Content-Type: text/html, which also loses its Content-Length — the signature of a component that streams the body rather than passing it through. Something is still parsing my HTML at the edge and finding nothing to change.

I do not know which feature it is. I have a guess and I am not going to print it as a cause, because a guess printed as a cause is exactly the failure this whole project exists to avoid.

The practical consequence: conditional requests remain unmeasurable on HTML, and measurable on everything else.

A short note about names

While investigating, I asked the CDN's own dashboard assistant what was injecting content into my pages. It told me, more than once, that the feature I was describing did not exist in my dashboard or in the current documentation.

At the time I accepted that and withdrew the name I had inferred from the behaviour.

Then I fetched the injected URL, and the page it returned had Labyrinth in its title.

I do not think anyone was lying to me. I think a support system genuinely could not find a feature that was running on the zone it was inspecting. But the sequence is worth sitting with, because it is this project's entire thesis arriving from an unexpected direction:

A vendor's self-report about its own product is not evidence about that product's behaviour.

I wrote that sentence about language models. It turned out to be about my own infrastructure.

What I am not claiming

Every one of these is a documented product feature, not a defect. Some are things a lot of site owners actively want — I can easily imagine wanting a decoy page for unidentified scrapers, and plenty of publishers very much want Disallow: / in front of AI training crawlers. That is a legitimate choice.

I am not claiming malice. I am not claiming anyone was hiding anything.

I am claiming three narrower things:

I did not switch these on, and I did not know they were on. They were defaults on a zone I had barely configured.
They changed what I published, in a direction opposite to my stated intent, on the one property where that intent was the entire point.
Nothing inside my application could have detected it. The logs were clean. The tests passed. The server sent the right bytes every time. The alteration happened after the last line of code I control.

And one thing I genuinely cannot establish: when it started. My origin only records what it sent. The CDN's additions have no history on my side. They were present for the whole investigation and may have been present since the day the zone was created. I am not going to date them, and you should distrust anyone who dates theirs.

The full check

You ran the first one at the top. Here are the other three.

# Is something prepended to your robots.txt?
curl -s https://your-domain/robots.txt | grep -c "Cloudflare Managed"

# Is something injected into your HTML?
curl -s https://your-domain/ | grep -c 'cdn-cgi/content'

# Does your validator survive the trip?
curl -sI https://your-domain/ | grep -i '^etag:'

And the one that found all of it — fetch a page from your origin and through your CDN, and diff the bytes. If they differ and you did not ask them to, that difference is being served to every reader you have, including the machines that decide what your site is about.

The part that generalises

I spent three days building an instrument to check whether AI systems tell the truth about what they read. The instrument worked. It just took me until day three to point it at the thing standing between me and everyone I was measuring.

The lesson has nothing to do with any particular CDN, and I would have found the same class of problem behind any of them:

A verification whose evidence comes from the party being checked is not a verification.

We have quietly accepted an arrangement in which almost everything we believe about our own systems is self-reported by those systems. The application says it served the right bytes. The dashboard says the setting is off. The support assistant says the feature does not exist. The analytics panel says these are the pages people want. The model says it does not remember your site.

Each of those is a party describing its own conduct. Not one of them is evidence. And they are not usually wrong — which is exactly what makes them dangerous, because a source that is right ninety-nine times teaches you to stop checking on the hundredth.

The gap between what you publish and what is received is not a CDN problem. It is the general shape of the thing: every layer you did not write is a layer that can answer for you, and every layer you did not write will tell you it did not.

There is only one class of evidence that does not come from an interested party — an independent observation, taken from outside, dated, and repeatable by someone who does not trust you either.

So the practical version of all this is much smaller than the principle, and you can do it this afternoon: fetch your own site from outside, compare it to what you meant to send, and write down the date you looked.

The only thing that is not a self-report is the bytes, measured from outside, by you.

The full technical record — every measurement, every reproduction command, and what this does and does not falsify — is committed at docs/cdn-interventions.md.

The lab page carries every figure quoted above, live. Those counts can be inflated: anyone can send requests claiming to be GPTBot, and this site does not refuse them, because refusing traffic would change what the instrument can measure. What cannot be inflated is the corroborated column — reaching it requires sending from the vendor's own published addresses. So if the injected Disallow: / was suppressing genuine crawlers, you will be able to watch that number move over the coming weeks without taking my word for any of it. That is the only kind of proof this article is entitled to offer.

If you run the checks on your own domain and find something, I would like to hear about it.

The code here was written with Claude, on an architecture and a set of rules that predate it. What to measure, what could be concluded from it, and what had to be thrown away were my calls — the repository's history shows which is which, including where I was wrong.