AI crawler

An automated client that fetches web pages on behalf of an AI system.

They divide roughly into three kinds, and the distinction matters because they arrive under different conditions and for different reasons.

Corpus crawlers

These collect pages in bulk to build training or index corpora. They arrive on their own schedule, unprompted by any user, and they tend to sweep broadly. What they take may influence a model long after the visit.

Retrieval crawlers

These maintain a search index that an AI product queries when answering. They behave much like traditional search crawlers, but the index they build feeds generated answers rather than a list of links.

User-triggered fetchers

These fetch a page because a person, right now, asked an assistant something that required reading it. They arrive one URL at a time, correlate with human activity patterns, and are the closest thing to a referral in the AI era.

A crawler declares which it is through its User-Agent string. That declaration is unverified — anyone can send any string — so this site records it as a claim and reasons about behaviour separately. See what we measure.