What CrawlCheck collects, and how to opt out
In one sentence
CrawlCheck reads public URLs only (robots.txt, sitemaps, llms.txt, entitymap.json, agents.md and the requested page) and never signs in or submits a form. Machine-file requests identify themselves as CrawlCheck/1.0 and are signed; the homepage is also fetched once under each named crawler user-agent from CrawlCheck’s own address, which is the measurement. It keeps a shareable report (deletable by the domain owner) and one measurement row per domain, publishes aggregate statistics without naming domains, names domains in the directory, the registry, the census frame, the daily snapshot of signed answers and the dataset files, and honours a CrawlCheck disallow or the line CrawlCheck: none in robots.txt on the next scan: the domain leaves the dataset, the registry and the crawl queue, and a directory vendor stays listed as opted out with no score.
If our crawler reached your server and you want to know what it is, this is the page.
~30
read-only requests per scan
1
owner request deletes what we keep
0
logins, forms, private pages
1
line in robots.txt opts you out
Who is fetching
Requests carry the user-agent Mozilla/5.0 (compatible; CrawlCheck/1.0; +https://crawlcheck.io/bot). Every request that fetches your machine files and control paths identifies itself this way. The measurement itself is the one exception: to see what each crawler is served, a scan also asks for your homepage once under each of 16 named crawler request profiles (15 crawler names: GPTBot, ClaudeBot, Googlebot as desktop and as smartphone, and the rest), from our own address. When a scan also crawls the site, up to 3 inner pages that answered normally are asked for once more under each search and AI crawler name, so a server that refuses a crawler only on an inner page is seen. The tools that compare what crawlers receive (the agent lens, the freshness check) send a few of those names to the one public page they are given. That address is not in any operator’s published range, so a site that verifies crawlers properly will refuse those requests, and the report says so. We never use those strings to get past a block, and we send them only to the public pages a scan or a tool was asked to read: no form, no login, nothing behind authentication.
Signed, so the name can be checked. Every request we send under that user-agent carries an Ed25519 signature (HTTP Message Signatures, the Web Bot Auth profile) over your hostname, and a Signature-Agent header naming this site. The public keys are at /.well-known/http-message-signatures-directory, and that response is itself signed by every key it lists. What the bot is and how it behaves is at /.well-known/signature-agent-card; key changes are recorded at /.well-known/security-key-rotation.json. We own no IP range — requests leave through a shared edge network — so /.well-known/ips.json lists none: an address is never proof a request was ours, the signature is. Check any signed request, ours or anyone’s, with the signature checker.
One exception, stated plainly because it matters: when a report measures what each agent receives, we deliberately send those user-agent strings to your public URL and record what comes back. Those requests come from our own address, which is not in the range those operators publish, so a site that verifies crawlers properly should refuse them. That is a correct result and the report says so. Those probes are sent unsigned: a signature would let a site that verifies Web Bot Auth recognise the probe as us and serve it something the real crawler never receives, and the measurement would stop measuring.
What we read
- Public URLs only. We never sign in, never submit a form, and never fetch anything behind authentication.
robots.txt,sitemap.xml,llms.txt,entitymap.json,agents.md,/.well-known/ucp, and the page requested. A crawl, when one runs, also reads public pages it reaches from your sitemap and internal links.- Two made-up paths per host (
/cc-control-<random>.txtat the root, and the same name one directory deep), so a server that answers every address with a page is not mistaken for one that has the file, and a rule that only guards a subdirectory is visible. - The delivered HTML, response headers and status codes for that page.
What we keep
- A shareable report at
/r/<id>, kept indefinitely: 90 days in the fast store, then from the sealed archive, because receipts, disputes and the daily seal point at it. A report started as unlisted is sealed and archived, opens from its link, and stays out of the registry, datasets, history and Resolve. A private one is not archived and does not open from the link: it opens only for the account that holds the site’s domain claim (proof of control: a DNS TXT record, a file at /.well-known/crawlcheck-domain.txt, a homepage meta tag, or a Search Console property the account owns) and the operator; listing a domain on a licence is not enough. - The bytes each crawler identity received, kept as evidence for as long as the retention table says, so a proof can be re-checked. They are not published: the bytes behind a finding are shown only inside that site's report, where a plan, the trial or the fix service opens them, and anyone can confirm a stored copy still hashes to its SHA-256. After the bytes are deleted the hashes and the day's anchor remain, so what was served stays provable but can no longer be replayed. Bundles are scanned for keys, tokens and private addresses before any section is served.
- Deletion. The verified owner of a domain can delete everything kept about it, stored bytes included, archive included, at /manage/data; a signed deletion record states what was removed. The daily seal keeps only a hash, which reveals nothing once the record is gone. A robots.txt opt-out hides archived reports at once.
- One row per domain holding the measurements and scores, so a re-scan can show what changed.
- Aggregate counters behind the dataset page.
- An email address, if you gave one before a scan. The site no longer asks for one. It is kept to send CrawlCheck product updates until you unsubscribe: every message carries a one-click stop link, and a reply asking to be removed is enough.
The rates and counts on the dataset page are aggregates and name no domain. Three published sets do name domains: the census lists the sites in its frame, the daily snapshot carries each measured domain’s signed answer, and the dataset files hold one row per measured site (a 25-row sample is free; the files are licensed). A domain that opts out leaves all three.
What is deleted, store by store
Two routes: a robots.txt opt-out, and deletion by the verified owner at /manage/data or through a removal request.
| What is kept | A robots.txt opt-out | The owner deletes the domain |
|---|---|---|
| The latest measurement, the registry row, score, finding and fact history, the fix ledger and the daily readings | Deleted by the scan that reads the opt-out | Deleted |
| Archived records and the bytes each crawler received | Deleted by that scan | Deleted |
| The signed answer served for the domain | Deleted; an answer then says the site opted out | Deleted, with every archived answer |
A report link already issued (/r/<id>) | The archived copy is deleted; the fast copy expires within 90 days | Deleted |
| The other measurements: lab (PageSpeed) results, entity readings, listing and business-record readings, answer-visibility summaries, analytics days, certificate history and the history of signed answers | Deleted by that scan | Deleted |
| What the owner or licence holder created: receipts, disputes, form and CRM settings, recorded workflows and goals | Kept until the owner deletes them | Deleted |
| The daily proof digest (a hash sealed with every other site measured that day) | Kept | Kept: a hash, not data; removing it would break the other sites’ proofs |
| The opt-out marker | Kept, so the site is not scanned again | Kept if one was declared |
| Anonymous totals already published | Not recomputed; they name no domain | Not recomputed; they name no domain |
| The licence and account | Not affected | Kept; remove the domain from the licence at /manage |
| Backups inside the account (twice a week, the last 7 kept) | A copy of the deleted rows stays until the backup ages out, about 25 days | The same |
| The encrypted off-account backup | Ages out after 7 days | The same |
| The sealed-day mirror (private repository: URL, status, size, type and hashes, never page text) | Dropped from every mirrored day on the next nightly run; earlier commits keep the old copy until the history is rewritten | The same |
| The deletion record | None is written | Kept: counts and digests only, signed, nothing that was removed |
Backups are read only to restore after data loss; they are not searched and nothing is served from them. Evidence bytes are not in the backups, so they are gone when they are deleted.
Google accounts you connect
Two features read data from Google on your behalf, and only after you sign in and grant them. Neither runs unless you do.
- Search Console. With the
webmasters.readonlypermission we read clicks, impressions, click-through rate and average position per property per day. Read-only; we cannot change anything in Search Console. - Business Profile. With the
business.managepermission we read the profiles your account manages — name, phone, address, website, hours, category — and their daily calls, direction requests, website clicks, impressions and monthly search keywords. Google grants that permission as read-and-write. We read by default and write only when you tell us to. The licence holder or the owner can send a correction to the profile’s website or phone, or to its service-area places, through the fixes API. Every such request is a dry run unless it carriesconfirm: APPLY; a website or phone can only be set to a value your own site publishes; name, address and categories are never written; and each write leaves a signed receipt with the value before, the value sent and the value read back.
What we keep: a refresh token stored against your licence, the profile fields above, and the daily and monthly series. The token is never shown on any page or API. Disconnect on the licence page deletes the token and every series read with it. Where Google reports a keyword count only as “below a threshold”, we store the threshold and say so; we do not turn it into a number.
If page text is ever sent to a third party for analysis — the content-entity check is designed to use Google’s Natural Language API — that check will be named here before it is switched on, with what is sent and what comes back. As of this page’s date it is not on, and nothing a scan reads leaves this service.
Requests to this site
Separate from scanning, and stated because a page listing what we keep should list all of it: this site keeps a short operational record of requests made to it, the way any web server keeps an access log. It holds the last 200 requests and a per-address count that expires after eight days, and it is readable only by us.
- None of it is published. No public page and no public API reads it. Where a visitor address does appear in something public — the crawler figures on the dataset page and at
/api/telemetry— it is reduced to a network, a /24 or a /48, never a full address. - Assets are not recorded as entries. A favicon, stylesheet, script, share image or badge is never one of the 200 entries: one fetched by a browser is not recorded at all, and one fetched by any other client only adds to that client’s line in the daily summary when it already has one.
- Our own requests. Our own signed requests are recorded, filed as ours and kept out of public counts; a request that only says it is ours, with a header or a CrawlCheck name in its user agent, is recorded like any other.
- Query strings are never stored, so a licence key in a URL cannot end up in that record.
- What each of the 200 entries holds: the time, address, method, path, user agent, referrer, the network’s owner, number, country and city, the status the request was answered with, the name the client gives itself, whether it looks like a browser, whether the path is one
robots.txtdisallows or another marked path, whether the client is on a verified-crawler list and, when a request is signed, whether the signature verified and whose key signed it. Requests that were refused or found nothing are kept the same way as requests that were served. - robots.txt, llms.txt and llms-full.txt fetched by a client that is not a browser are counted in the daily summary below (how often, and whether that client read robots.txt that day), within its 2,000-line and nine-day bounds; they are not among the 200 entries. A favicon, stylesheet, script, share image or badge fetched by such a client only adds to its count when it already has a line.
- A daily summary of automated requests and refused attempts is built from those entries so that a crawler or script can be named: one line per client name and network, with the user agent, the network’s owner, number, country and city, counts, the statuses it received, up to six paths, the path marks above, the signature result and the last address seen. It holds at most 2,000 lines a day and each day is deleted within nine days. To put a name to a client we may look up public DNS records for its address; a public DNS resolver receives the address for that lookup. At most 1,000 lookup notes are kept, each dropped 30 days after the lookup; a note can hold the DNS name found, which can itself contain the address. A browser-style request that was served is counted in that summary and not listed, whatever the page. It is readable only by us, and nothing in it is used to block anyone.
- The llms.txt reader check on a site’s report shows, for each reader that followed the link, the name it claimed with the kind of network and the country it came from. The counts are kept for 90 days, with each day’s network owners and countries; no address is shown or kept in them.
Performance timings from this site’s own pages
These pages load a small script that measures how fast this site was for you — how long the main content took to appear, how quickly it answered your first interaction, and how much the layout moved — and sends one message when you leave the page. We run it on ourselves because we ask other people to run it, and it would be poor form to publish numbers we had not collected the same way.
- No cookie, no identifier, no profile. Nothing links one page view to another, or to you.
- The path only. Query strings and fragments are stripped before the message is sent, for the same reason they are kept out of the access log above.
- On this site only, the arrival message also carries coarse browser traits — whether the browser reports itself as automated, its time zone, first language, screen size and core count — so automated agents can be told apart from people. They stay with the visit for 30 minutes; only per-burst totals for automated traffic are kept longer, for 90 days.
- Timings, plus a coarse element name — which element was slowest to paint, for instance — with connection type and window width. Readings are merged into counts; individual messages are not kept.
- Blocking it costs you nothing. Any content blocker stops
/v.jsand every page behaves exactly the same.
How to opt out
Either of these stops future measurement, deletes the domain’s stored record, its history, its other measurements and its archived bytes (store by store), and removes it from the crawl queue, the registry and every public count from then on. In the directory the vendor’s name stays, marked opted out with the effective date and no score, so an absence never looks like an editorial exclusion. No email, no form, no account — you declare it in a file you already control and we honour it on the next scan: that scan deletes the stored record, its score history and the corpus records built from them. A total published before then is not recomputed — it is a number, not a list, and it names no one.
A group in robots.txt
User-agent: CrawlCheck
Disallow: /Or one line, anywhere in it
CrawlCheck: noneA page you request yourself can still be scanned — you may be the one asking — but nothing about it enters the dataset or any count. The report you asked for is kept for its link like every report, and is deleted with the rest of the domain’s records on the owner’s request.
The directory
The directory publishes a score for named companies, which is different from every other measurement here, and it runs under stated rules.
- No paid placement, ever. No vendor is listed for payment, no listing is removed for payment, and no score is edited on request. Comparison guides are separate: a product can pay a one-time fee for a link and a badge there, labelled as paid, and the fee never changes a result. Nothing a licence unlocks changes a measurement anywhere on this site, and a directory that broke that rule would make every other number here worth less.
- Every score links to its evidence. Each row opens the full report: the status codes, headers and bytes each identity received, on a stated date, under a stated score version.
- A dispute earns a re-scan, not an edit. Tell us what is wrong (or email hello@crawlcheck.io) and the site is measured again; the new reading is published as measured. If our instrument was wrong, the correction is published in the findings, as it has been before.
- The opt-out binds here too. The robots.txt line below removes a domain’s measurements from the dataset and the counters. The directory listing itself remains, marked as opted out with no score, so an absence is visible rather than looking like an oversight.
- Categories are ours and are unscored. They describe what a product is primarily sold as. No pricing is published, because a price we did not measure is a claim we cannot stand behind.
What we do not do
Fetch private pages
Nothing authenticated, nothing behind a login.
Sell raw records
Raw scan records are not sold. Named records appear only where this page says: the directory, the registry, the census frame, the daily snapshot and licensed lane files.
Name domains in aggregates
Not in the aggregate statistics (/data and the rates quoted across this site). The directory, the registry, the census frame, the daily snapshot of signed answers and the dataset files do name domains, as described above, and the opt-out below removes a domain from each.
Impersonate to get past a block
Another operator’s crawler name is never used to get around a refusal.
Removal requests
- Who can ask
- Anyone. Whoever controls the site proves it on the site itself; a person whose own data appears in a report proves the email address they give; anything else (legal, safety) is proven the same way. You do not need a CrawlCheck account or licence.
- How it is verified
- Site control: a token served at /.well-known/crawlcheck-removal.txt on the domain, or a TXT record at _crawlcheck-removal.<domain>. Personal data and other requests: a 6-digit code mailed to the address given, valid for 7 days.
- Timeframe
- Proven site control is acted on at once: everything held about the domain is deleted and, if asked, the domain is opted out of future scans. Other requests are decided within 30 days of confirmation.
- What can be removed
- The named text, from every stored copy (reports, history, evidence and the page bytes a scan received); a single record; or everything held about a domain. Factual disagreements about a measurement go to a dispute instead, where the evidence is re-checked.
- Appeal
- Every decision can be appealed once, within 30 days, with the request id and the token given when it was made. The appeal is decided within 30 days and that decision is final.
- What stays
- The record's digest inside the day's sealed proof: a hash, anchored in Bitcoin together with every other site measured that day, which cannot be read back into the text. Removing it would break the proof for every other site. One signed removal record says what was done, in counts and digests, with no removed text and nothing about who asked.
- Holds
- A request stops publication at once. Only a hold delays the erasure itself: an abuse investigation, a legal preservation demand, or evidence for a dispute filed by someone else. A hold lasts at most 90 days unless renewed with a reason; when it ends, the waiting request runs on its own. A site's own open dispute never blocks its own deletion: deleting withdraws it. Holds are counted at /api/holds, never listed by domain.
- Your own data in the request
- The address you give and the text you ask to remove are used only for this request and deleted 30 days after the final decision. Unconfirmed requests are deleted after 7 days.
