Should you block AI crawlers?
It depends on whether you want your content to be retrievable in AI answers, and the two cases are increasingly distinguishable. Blocking corpus crawlers keeps content out of training; blocking retrieval crawlers keeps you out of live answers, which is closer to being delisted from search. This observatory allows everything, because being read is the subject of the study.
Context, not a recommendation.
This site is not the right source for a recommendation — it allows every crawler because observing them is the entire purpose. But the decision is frequently framed badly, and the framing is worth separating from the choice.
Different crawlers, different consequences
Blocking a corpus crawler reduces the chance of your text entering a training set. The effect is delayed and effectively permanent in either direction: content already collected is not withdrawn by blocking later.
Blocking a retrieval crawler keeps you out of answers being generated right now. That is much closer to removing yourself from a search index, and the cost is immediate.
Blocking a user-triggered fetcher means that when a person explicitly asks an assistant to read your page, it cannot. This is the closest thing to refusing a referred visitor.
Many operators publish separate agents for these roles, which makes a mixed policy possible: decline the corpus collection, permit the retrieval. Whether the separation is honoured is itself an empirical question.
What blocking does not do
robots.txt has no enforcement. It expresses a preference to clients that choose to read it. A client that ignores it is unaffected, and content already indexed elsewhere remains reachable regardless.
A practical note
Some infrastructure providers now block AI crawlers by default on new domains. If you have not deliberately allowed them, you may already be blocking them without having chosen to — worth verifying by requesting your own site with a crawler user agent and observing the response code.
What the record shows today
224 external requests have declared one of these identities. Checked against the address ranges each vendor publishes for its own crawler:
| Outcome | Requests | What it means |
|---|---|---|
| Corroborated | 74 | came from inside the vendor's published range |
| Vendor's other range | 0 | a different range belonging to the same vendor |
| Contradicted | 30 | the vendor publishes a list and this address is not on it |
| Uncheckable | 120 | no machine-readable list exists to check against |
Only contradicted is evidence against a client, and it means the declaration is uncorroborated — never intent. Uncheckable is a gap in a vendor's publishing: Anthropic and Common Crawl publish nothing verifiable, so every one of their agents lands there however genuine it is. Snapshot captured 2026-07-26. The per-agent split.
Computed from the record at 2026-07-30 03:32:52 UTC. Every figure here is recomputed when the page is requested, so this section moves as the record does — week by week.