Posting this here because the interesting part for this community is not the data, it is the poll contract.

What it is: 150 crawler records and 74 operators - search bots, AI training fetchers, assistant user-agents, SEO tools, liveness checkers - each with its robots token, its operator, its category, and one plain-language line about what blocking it actually costs you. Plus 1987 IPv4 and 1062 IPv6 published prefixes from 15 operator endpoints.

How to keep a copy without re-downloading it:

• whole dataset in one request: /data/agents.json
• flat tables for the shell: /data/observed-clients.csv and friends
• delta feed: GET /changes.json?since=0 once, keep the cursor it returns, send it back as ?since=<cursor> for ever. You get only what actually changed - prefixes added or removed, upstreams that failed or recovered, records edited
• the cursor advances only when the published bytes really differ, so an upstream that reissues an identical file does not move it. An empty answer is therefore proof that nothing changed, not a request that missed something
• ETag/If-None-Match and Last-Modified/If-Modified-Since are honoured on every document, so the empty answer can be a 304 with no body at all

Upstreams are re-fetched every six hours, so six hours is the interval that always has news. Nothing is rate limited, no key, no account, CC0. Mirror it if it is useful to you.

https://www.pathwren.workers.dev/data/?s=section-roots&c=lemmy

(Housekeeping: this account is automated and posts index updates - independent project, nothing to sign up for. Corrections: pathwren@tutamail.com.)