Do AI crawlers respect robots.txt?
Compliance is a claim until it is measured on your own domain. robots.txt is advisory — it has no enforcement mechanism — so the only way to know which clients follow it is to publish a disallowed path that serves ordinary content and record who fetches it anyway. This site does exactly that with three separate disallowed paths.
Measuring — the record is too new to report rates.
robots.txt is advisory. It is a text file expressing a preference, with no enforcement behind it. A crawler that ignores it is not breaking anything technical; it is simply not doing what it was asked.
Published compliance claims mostly come from the crawler operators themselves, which is exactly the kind of self-report this observatory refuses. The alternative is to make compliance observable on a domain you control.
How this site measures it
Three paths are listed under Disallow in robots.txt: /internal/, /no-crawl/ and /private-preview/. Each serves an ordinary page returning 200. Nothing there is sensitive, hidden or trapped — they are normal pages that simply ask not to be crawled.
They are named differently on purpose. A path called /private-preview/ implies something worth having; /no-crawl/ does not. If the rate of ignored rules differs between them, the name is doing work.
Three separate prefixes also distinguish blanket behaviour from per-rule behaviour, since some clients honour the first matching group and disregard the rest.
What a violation does and does not prove
A fetch of a disallowed path proves the fetch happened. It does not prove the client read robots.txt and chose to disregard it — the client may never have requested the file. Separating those two cases requires seeing a robots.txt fetch from the same client beforehand, which is only reliable when the client is consistent about its address and declared identity.
Both figures are reported separately for that reason: fetches of disallowed paths, and fetches of disallowed paths by clients that had already read the rules.
Current state
Live counts are on the lab page. No compliance rate is published yet because the sample cannot support one.
What the record shows today
1 fetches of disallowed paths, from 1 distinct declared identities. 18 distinct user agents have requested robots.txt.
| User agent (claimed) | Fetches | Disallowed paths |
|---|---|---|
| Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 | 1 | 1 |
A fetch proves the fetch happened. It does not prove the client read the rules and chose to disregard them — it may never have requested the file.
1 fetches is not enough to publish a rate. This is a record of what happened, not a finding about crawler behaviour in general.
Computed from the record at 2026-07-30 04:27:30 UTC. Every figure here is recomputed when the page is requested, so this section moves as the record does — week by week.