Questions
Questions about how AI systems read the web, answered from what can be observed — and marked plainly where the record is not yet good enough to answer.
Several of these have no reliable public answer, which is the reason this observatory exists. Where a question is still being measured, the page says so and describes the instrument rather than inventing a figure.
Do AI crawlers respect robots.txt?
Compliance is a claim until it is measured on your own domain. robots.txt is advisory — it has no enforcement mechanism — so the only way to know which clients follow it is to publish a disallowed path that serves ordinary content and record who fetches it anyway. This site does exactly that with three separate disallowed paths.
Measuring — the record is too new to report rates.
Which AI crawlers execute JavaScript?
Most do not, but the assumption is rarely tested. A page whose content is injected by script and which then requests a beacon URL turns this into an observation: any client that reaches the beacon has demonstrably executed JavaScript, and any client that fetches the page without the beacon demonstrably has not.
Measuring — the JS probe is live.
How long does it take for newly published text to appear in an AI model?
There is no reliable public figure for this, largely because measuring it requires knowing that a specific page was ingested rather than guessing. Publishing a coined string that exists nowhere else, recording the instant it was published, and watching for that exact string to appear turns the interval into an observation.
Measuring — no appearance observed yet.
How can you tell whether an AI system has read your site?
Two independent sources of evidence: your own server logs, which show which crawlers fetched what and when, and a canary string published at a known instant whose later appearance proves ingestion. Server logs prove fetching; canaries prove ingestion. Asking a model proves nothing.
Method published; both instruments running.
Why not just ask the model what it knows about your site?
Because a model describing its own knowledge is the system under test giving evidence about itself. It can assert familiarity with pages it never read and deny ones it did, and nothing in the answer distinguishes the two. No layer may validate itself using its own output as independent evidence.
Founding principle of this observatory.
Should you block AI crawlers?
It depends on whether you want your content to be retrievable in AI answers, and the two cases are increasingly distinguishable. Blocking corpus crawlers keeps content out of training; blocking retrieval crawlers keeps you out of live answers, which is closer to being delisted from search. This observatory allows everything, because being read is the subject of the study.
Context, not a recommendation.
Which content format do AI crawlers fetch most?
Widely asserted, rarely measured. Publishing identical content as HTML, JSON, Markdown, plain text and RSS — each carrying a distinct marker — turns format preference from folklore into a count of which variants were actually fetched, and by whom.
Measuring — five format variants live.
What are GPTBot, ClaudeBot, PerplexityBot and CCBot?
Declared identities used by automated clients associated with AI systems, spanning corpus collection, retrieval indexing and user-triggered fetching. The names appear in the User-Agent header, which is unverified — the string is a claim about identity, not proof of it.
Reference; observed arrivals listed on the lab page.
Method
Answers here are constrained by the constitution: no figure is published that cannot be recomputed from the stored record, and no model's account of its own behaviour is treated as evidence. Live counts are on the lab page, and the ways they could mislead are listed in the methodology.