Which content format do AI crawlers fetch most?
Widely asserted, rarely measured. Publishing identical content as HTML, JSON, Markdown, plain text and RSS — each carrying a distinct marker — turns format preference from folklore into a count of which variants were actually fetched, and by whom.
Measuring — five format variants live.
Advice about structuring content for AI systems circulates well ahead of the evidence for it. Recommendations to prefer one markup style, add a particular file, or restructure pages are usually offered without a measurement behind them.
How this site measures it
The same statement is published five ways, each carrying its own coined marker:
- HTML — ordinary server-rendered markup
- JSON — structured, machine-first
- Markdown — the format most model training pipelines normalise toward
- Plain text — no markup at all
- RSS — the syndication path
All five are linked from the home page and listed in the sitemap, so no variant is easier to discover than another. Which get fetched, by which clients, and how often is then a matter of record.
An llms.txt file is also served. Whether anything fetches it is a finding in itself, given how often it is recommended.
What this can establish
Fetch counts per variant, per declared client. It cannot establish which format was preferred in any deeper sense — a crawler may fetch a format and discard it — nor can one small domain settle the question for the web. Distinct markers per variant do, however, make it possible to tell later which version a model saw.
Current state
Counts appear on the lab page under format preference. The sample is currently too small to interpret.
What the record shows today
The same content is published in six shapes at six addresses, all listed in the sitemap and none preferred by any link. Fetches so far:
| Format | Fetches | Distinct agents |
|---|---|---|
| probe_llms_txt | 14 | 9 |
| HTML requiring JavaScript | 10 | 7 |
| probe_text | 7 | 5 |
| probe_markdown | 6 | 5 |
| probe_json | 6 | 5 |
| HTML | 6 | 5 |
| probe_feed | 5 | 4 |
| HTML with a noscript fallback | 3 | 3 |
Reach is not equal between them — the sitemap lists all six, but nothing else on the site links them evenly, so a difference here may be a difference in discovery rather than in preference.
Computed from the record at 2026-07-30 04:25:50 UTC. Every figure here is recomputed when the page is requested, so this section moves as the record does — week by week.