What are GPTBot, ClaudeBot, PerplexityBot and CCBot?
Declared identities used by automated clients associated with AI systems, spanning corpus collection, retrieval indexing and user-triggered fetching. The names appear in the User-Agent header, which is unverified — the string is a claim about identity, not proof of it.
Reference; observed arrivals listed on the lab page.
These names appear in the User-Agent header of requests to public websites. Grouping them by what they are for is more useful than listing them, because the purposes carry different consequences for a site owner.
By role
Corpus collection. Agents gathering pages in bulk for training or for a general-purpose archive. They arrive unprompted and sweep broadly. CCBot, operated by the non-profit Common Crawl, belongs here and long predates the current generation of AI systems; its archives are widely reused.
Retrieval indexing. Agents maintaining an index that an AI product queries while answering a question. Behaviourally close to a traditional search crawler.
User-triggered fetching. Agents fetching one page because a person just asked an assistant something requiring it. Operators increasingly separate this role into its own declared name, which is what makes a mixed allow-some, decline-others policy expressible at all.
The important caveat
A user agent string is unverified. Any client can send any string, including one belonging to a well-known crawler. Counts by name are counts of claims.
Some operators publish IP ranges or reverse-DNS conventions that allow a declared identity to be checked, and where that exists it is worth using. Absent verification, this observatory stores the declaration verbatim, treats it as a claim, and reasons about observed behaviour separately.
Naming changes
Agent names, roles and the split between them change as operators revise their crawling. Any list is a snapshot. The reason this site records complete request headers rather than only a parsed agent name is so that the record stays interpretable when the names move.
Observed here
Declared clients actually seen by this site, with first and last appearance, are listed on the lab page.
What the record shows today
219 external requests have declared one of these identities. Checked against the address ranges each vendor publishes for its own crawler:
| Outcome | Requests | What it means |
|---|---|---|
| Corroborated | 74 | came from inside the vendor's published range |
| Vendor's other range | 0 | a different range belonging to the same vendor |
| Contradicted | 30 | the vendor publishes a list and this address is not on it |
| Uncheckable | 115 | no machine-readable list exists to check against |
Only contradicted is evidence against a client, and it means the declaration is uncorroborated — never intent. Uncheckable is a gap in a vendor's publishing: Anthropic and Common Crawl publish nothing verifiable, so every one of their agents lands there however genuine it is. Snapshot captured 2026-07-26. The per-agent split.
Computed from the record at 2026-07-30 03:31:51 UTC. Every figure here is recomputed when the page is requested, so this section moves as the record does — week by week.