close
CrawlCheck
Policy

What CrawlCheck collects, and how to opt out

In one sentence

CrawlCheck reads public URLs only (robots.txt, sitemaps, llms.txt, entitymap.json, agents.md and the requested page) and never signs in or submits a form. Machine-file requests identify themselves as CrawlCheck/1.0 and are signed; the homepage is also fetched once under each named crawler user-agent from CrawlCheck’s own address, which is the measurement. It keeps a shareable report (deletable by the domain owner) and one measurement row per domain, publishes aggregate statistics without naming domains, names domains in the directory, the registry, the census frame, the daily snapshot of signed answers and the dataset files, and honours a CrawlCheck disallow or the line CrawlCheck: none in robots.txt on the next scan: the domain leaves the dataset, the registry and the crawl queue, and a directory vendor stays listed as opted out with no score.

If our crawler reached your server and you want to know what it is, this is the page.

Everything one scan sends to your server
Named crawler user-agentshomepage only, once each16Browser and client controlshomepage, to compare against3As CrawlCheck, signedrobots.txt, llms.txt, two sitemaps, entity map × apex and www10Made-up control pathstwo per host, to see if the server answers everything433 GET requests, read-only, from one address, sent together — one scan. Watch and other scheduled re-reads repeat it on their own schedule.

~30

read-only requests per scan

1

owner request deletes what we keep

0

logins, forms, private pages

1

line in robots.txt opts you out

Identity

Who is fetching

Requests carry the user-agent Mozilla/5.0 (compatible; CrawlCheck/1.0; +https://crawlcheck.io/bot). Every request that fetches your machine files and control paths identifies itself this way. The measurement itself is the one exception: to see what each crawler is served, a scan also asks for your homepage once under each of 16 named crawler request profiles (15 crawler names: GPTBot, ClaudeBot, Googlebot as desktop and as smartphone, and the rest), from our own address. When a scan also crawls the site, up to 3 inner pages that answered normally are asked for once more under each search and AI crawler name, so a server that refuses a crawler only on an inner page is seen. The tools that compare what crawlers receive (the agent lens, the freshness check) send a few of those names to the one public page they are given. That address is not in any operator’s published range, so a site that verifies crawlers properly will refuse those requests, and the report says so. We never use those strings to get past a block, and we send them only to the public pages a scan or a tool was asked to read: no form, no login, nothing behind authentication.

Signed, so the name can be checked. Every request we send under that user-agent carries an Ed25519 signature (HTTP Message Signatures, the Web Bot Auth profile) over your hostname, and a Signature-Agent header naming this site. The public keys are at /.well-known/http-message-signatures-directory, and that response is itself signed by every key it lists. What the bot is and how it behaves is at /.well-known/signature-agent-card; key changes are recorded at /.well-known/security-key-rotation.json. We own no IP range — requests leave through a shared edge network — so /.well-known/ips.json lists none: an address is never proof a request was ours, the signature is. Check any signed request, ours or anyone’s, with the signature checker.

One exception, stated plainly because it matters: when a report measures what each agent receives, we deliberately send those user-agent strings to your public URL and record what comes back. Those requests come from our own address, which is not in the range those operators publish, so a site that verifies crawlers properly should refuse them. That is a correct result and the report says so. Those probes are sent unsigned: a signature would let a site that verifies Web Bot Auth recognise the probe as us and serve it something the real crawler never receives, and the measurement would stop measuring.

Scope

What we read

  • Public URLs only. We never sign in, never submit a form, and never fetch anything behind authentication.
  • robots.txt, sitemap.xml, llms.txt, entitymap.json, agents.md, /.well-known/ucp, and the page requested. A crawl, when one runs, also reads public pages it reaches from your sitemap and internal links.
  • Two made-up paths per host (/cc-control-<random>.txt at the root, and the same name one directory deep), so a server that answers every address with a page is not mistaken for one that has the file, and a rule that only guards a subdirectory is visible.
  • The delivered HTML, response headers and status codes for that page.
Retention

What we keep

  • A shareable report at /r/<id>, kept indefinitely: 90 days in the fast store, then from the sealed archive, because receipts, disputes and the daily seal point at it. A report started as unlisted is sealed and archived, opens from its link, and stays out of the registry, datasets, history and Resolve. A private one is not archived and does not open from the link: it opens only for the account that holds the site’s domain claim (proof of control: a DNS TXT record, a file at /.well-known/crawlcheck-domain.txt, a homepage meta tag, or a Search Console property the account owns) and the operator; listing a domain on a licence is not enough.
  • The bytes each crawler identity received, kept as evidence for as long as the retention table says, so a proof can be re-checked. They are not published: the bytes behind a finding are shown only inside that site's report, where a plan, the trial or the fix service opens them, and anyone can confirm a stored copy still hashes to its SHA-256. After the bytes are deleted the hashes and the day's anchor remain, so what was served stays provable but can no longer be replayed. Bundles are scanned for keys, tokens and private addresses before any section is served.
  • Deletion. The verified owner of a domain can delete everything kept about it, stored bytes included, archive included, at /manage/data; a signed deletion record states what was removed. The daily seal keeps only a hash, which reveals nothing once the record is gone. A robots.txt opt-out hides archived reports at once.
  • One row per domain holding the measurements and scores, so a re-scan can show what changed.
  • Aggregate counters behind the dataset page.
  • An email address, if you gave one before a scan. The site no longer asks for one. It is kept to send CrawlCheck product updates until you unsubscribe: every message carries a one-click stop link, and a reply asking to be removed is enough.

The rates and counts on the dataset page are aggregates and name no domain. Three published sets do name domains: the census lists the sites in its frame, the daily snapshot carries each measured domain’s signed answer, and the dataset files hold one row per measured site (a 25-row sample is free; the files are licensed). A domain that opts out leaves all three.

Deletion

What is deleted, store by store

Two routes: a robots.txt opt-out, and deletion by the verified owner at /manage/data or through a removal request.

What is keptA robots.txt opt-outThe owner deletes the domain
The latest measurement, the registry row, score, finding and fact history, the fix ledger and the daily readingsDeleted by the scan that reads the opt-outDeleted
Archived records and the bytes each crawler receivedDeleted by that scanDeleted
The signed answer served for the domainDeleted; an answer then says the site opted outDeleted, with every archived answer
A report link already issued (/r/<id>)The archived copy is deleted; the fast copy expires within 90 daysDeleted
The other measurements: lab (PageSpeed) results, entity readings, listing and business-record readings, answer-visibility summaries, analytics days, certificate history and the history of signed answersDeleted by that scanDeleted
What the owner or licence holder created: receipts, disputes, form and CRM settings, recorded workflows and goalsKept until the owner deletes themDeleted
The daily proof digest (a hash sealed with every other site measured that day)KeptKept: a hash, not data; removing it would break the other sites’ proofs
The opt-out markerKept, so the site is not scanned againKept if one was declared
Anonymous totals already publishedNot recomputed; they name no domainNot recomputed; they name no domain
The licence and accountNot affectedKept; remove the domain from the licence at /manage
Backups inside the account (twice a week, the last 7 kept)A copy of the deleted rows stays until the backup ages out, about 25 daysThe same
The encrypted off-account backupAges out after 7 daysThe same
The sealed-day mirror (private repository: URL, status, size, type and hashes, never page text)Dropped from every mirrored day on the next nightly run; earlier commits keep the old copy until the history is rewrittenThe same
The deletion recordNone is writtenKept: counts and digests only, signed, nothing that was removed

Backups are read only to restore after data loss; they are not searched and nothing is served from them. Evidence bytes are not in the backups, so they are gone when they are deleted.

Connected accounts

Google accounts you connect

Two features read data from Google on your behalf, and only after you sign in and grant them. Neither runs unless you do.

  • Search Console. With the webmasters.readonly permission we read clicks, impressions, click-through rate and average position per property per day. Read-only; we cannot change anything in Search Console.
  • Business Profile. With the business.manage permission we read the profiles your account manages — name, phone, address, website, hours, category — and their daily calls, direction requests, website clicks, impressions and monthly search keywords. Google grants that permission as read-and-write. We read by default and write only when you tell us to. The licence holder or the owner can send a correction to the profile’s website or phone, or to its service-area places, through the fixes API. Every such request is a dry run unless it carries confirm: APPLY; a website or phone can only be set to a value your own site publishes; name, address and categories are never written; and each write leaves a signed receipt with the value before, the value sent and the value read back.

What we keep: a refresh token stored against your licence, the profile fields above, and the daily and monthly series. The token is never shown on any page or API. Disconnect on the licence page deletes the token and every series read with it. Where Google reports a keyword count only as “below a threshold”, we store the threshold and say so; we do not turn it into a number.

If page text is ever sent to a third party for analysis — the content-entity check is designed to use Google’s Natural Language API — that check will be named here before it is switched on, with what is sent and what comes back. As of this page’s date it is not on, and nothing a scan reads leaves this service.

Access log

Requests to this site

Separate from scanning, and stated because a page listing what we keep should list all of it: this site keeps a short operational record of requests made to it, the way any web server keeps an access log. It holds the last 200 requests and a per-address count that expires after eight days, and it is readable only by us.

  • None of it is published. No public page and no public API reads it. Where a visitor address does appear in something public — the crawler figures on the dataset page and at /api/telemetry — it is reduced to a network, a /24 or a /48, never a full address.
  • Assets are not recorded as entries. A favicon, stylesheet, script, share image or badge is never one of the 200 entries: one fetched by a browser is not recorded at all, and one fetched by any other client only adds to that client’s line in the daily summary when it already has one.
  • Our own requests. Our own signed requests are recorded, filed as ours and kept out of public counts; a request that only says it is ours, with a header or a CrawlCheck name in its user agent, is recorded like any other.
  • Query strings are never stored, so a licence key in a URL cannot end up in that record.
  • What each of the 200 entries holds: the time, address, method, path, user agent, referrer, the network’s owner, number, country and city, the status the request was answered with, the name the client gives itself, whether it looks like a browser, whether the path is one robots.txt disallows or another marked path, whether the client is on a verified-crawler list and, when a request is signed, whether the signature verified and whose key signed it. Requests that were refused or found nothing are kept the same way as requests that were served.
  • robots.txt, llms.txt and llms-full.txt fetched by a client that is not a browser are counted in the daily summary below (how often, and whether that client read robots.txt that day), within its 2,000-line and nine-day bounds; they are not among the 200 entries. A favicon, stylesheet, script, share image or badge fetched by such a client only adds to its count when it already has a line.
  • A daily summary of automated requests and refused attempts is built from those entries so that a crawler or script can be named: one line per client name and network, with the user agent, the network’s owner, number, country and city, counts, the statuses it received, up to six paths, the path marks above, the signature result and the last address seen. It holds at most 2,000 lines a day and each day is deleted within nine days. To put a name to a client we may look up public DNS records for its address; a public DNS resolver receives the address for that lookup. At most 1,000 lookup notes are kept, each dropped 30 days after the lookup; a note can hold the DNS name found, which can itself contain the address. A browser-style request that was served is counted in that summary and not listed, whatever the page. It is readable only by us, and nothing in it is used to block anyone.
  • The llms.txt reader check on a site’s report shows, for each reader that followed the link, the name it claimed with the kind of network and the country it came from. The counts are kept for 90 days, with each day’s network owners and countries; no address is shown or kept in them.
RUM

Performance timings from this site’s own pages

These pages load a small script that measures how fast this site was for you — how long the main content took to appear, how quickly it answered your first interaction, and how much the layout moved — and sends one message when you leave the page. We run it on ourselves because we ask other people to run it, and it would be poor form to publish numbers we had not collected the same way.

  • No cookie, no identifier, no profile. Nothing links one page view to another, or to you.
  • The path only. Query strings and fragments are stripped before the message is sent, for the same reason they are kept out of the access log above.
  • On this site only, the arrival message also carries coarse browser traits — whether the browser reports itself as automated, its time zone, first language, screen size and core count — so automated agents can be told apart from people. They stay with the visit for 30 minutes; only per-burst totals for automated traffic are kept longer, for 90 days.
  • Timings, plus a coarse element name — which element was slowest to paint, for instance — with connection type and window width. Readings are merged into counts; individual messages are not kept.
  • Blocking it costs you nothing. Any content blocker stops /v.js and every page behaves exactly the same.
Opt out

How to opt out

Either of these stops future measurement, deletes the domain’s stored record, its history, its other measurements and its archived bytes (store by store), and removes it from the crawl queue, the registry and every public count from then on. In the directory the vendor’s name stays, marked opted out with the effective date and no score, so an absence never looks like an editorial exclusion. No email, no form, no account — you declare it in a file you already control and we honour it on the next scan: that scan deletes the stored record, its score history and the corpus records built from them. A total published before then is not recomputed — it is a number, not a list, and it names no one.

A group in robots.txt

User-agent: CrawlCheck
Disallow: /

Or one line, anywhere in it

CrawlCheck: none

A page you request yourself can still be scanned — you may be the one asking — but nothing about it enters the dataset or any count. The report you asked for is kept for its link like every report, and is deleted with the rest of the domain’s records on the owner’s request.

Named scores

The directory

The directory publishes a score for named companies, which is different from every other measurement here, and it runs under stated rules.

  • No paid placement, ever. No vendor is listed for payment, no listing is removed for payment, and no score is edited on request. Comparison guides are separate: a product can pay a one-time fee for a link and a badge there, labelled as paid, and the fee never changes a result. Nothing a licence unlocks changes a measurement anywhere on this site, and a directory that broke that rule would make every other number here worth less.
  • Every score links to its evidence. Each row opens the full report: the status codes, headers and bytes each identity received, on a stated date, under a stated score version.
  • A dispute earns a re-scan, not an edit. Tell us what is wrong (or email hello@crawlcheck.io) and the site is measured again; the new reading is published as measured. If our instrument was wrong, the correction is published in the findings, as it has been before.
  • The opt-out binds here too. The robots.txt line below removes a domain’s measurements from the dataset and the counters. The directory listing itself remains, marked as opted out with no score, so an absence is visible rather than looking like an oversight.
  • Categories are ours and are unscored. They describe what a product is primarily sold as. No pricing is published, because a price we did not measure is a claim we cannot stand behind.
Never

What we do not do

Fetch private pages

Nothing authenticated, nothing behind a login.

Sell raw records

Raw scan records are not sold. Named records appear only where this page says: the directory, the registry, the census frame, the daily snapshot and licensed lane files.

Name domains in aggregates

Not in the aggregate statistics (/data and the rates quoted across this site). The directory, the registry, the census frame, the daily snapshot of signed answers and the dataset files do name domains, as described above, and the opt-out below removes a domain from each.

Impersonate to get past a block

Another operator’s crawler name is never used to get around a refusal.

Removal requests

Who can ask
Anyone. Whoever controls the site proves it on the site itself; a person whose own data appears in a report proves the email address they give; anything else (legal, safety) is proven the same way. You do not need a CrawlCheck account or licence.
How it is verified
Site control: a token served at /.well-known/crawlcheck-removal.txt on the domain, or a TXT record at _crawlcheck-removal.<domain>. Personal data and other requests: a 6-digit code mailed to the address given, valid for 7 days.
Timeframe
Proven site control is acted on at once: everything held about the domain is deleted and, if asked, the domain is opted out of future scans. Other requests are decided within 30 days of confirmation.
What can be removed
The named text, from every stored copy (reports, history, evidence and the page bytes a scan received); a single record; or everything held about a domain. Factual disagreements about a measurement go to a dispute instead, where the evidence is re-checked.
Appeal
Every decision can be appealed once, within 30 days, with the request id and the token given when it was made. The appeal is decided within 30 days and that decision is final.
What stays
The record's digest inside the day's sealed proof: a hash, anchored in Bitcoin together with every other site measured that day, which cannot be read back into the text. Removing it would break the proof for every other site. One signed removal record says what was done, in counts and digests, with no removed text and nothing about who asked.
Holds
A request stops publication at once. Only a hold delays the erasure itself: an abuse investigation, a legal preservation demand, or evidence for a dispute filed by someone else. A hold lasts at most 90 days unless renewed with a reason; when it ends, the waiting request runs on its own. A site's own open dispute never blocks its own deletion: deleting withdraws it. Holds are counted at /api/holds, never listed by domain.
Your own data in the request
The address you give and the text you ask to remove are used only for this request and deleted 30 days after the final decision. Unconfirmed requests are deleted after 7 days.

Make a removal request

Questions about this page

QHow do I opt my domain out of CrawlCheck?
Add User-agent: CrawlCheck with Disallow: / to robots.txt, or the single line CrawlCheck: none anywhere in it. No email or form. The exclusion is honoured on the next scan: that scan deletes the stored record and its history, and the domain leaves the dataset, the registry and the crawl queue, and adds nothing to any public count after that. A vendor in the directory stays listed by name, marked opted out with no score, so an absence never looks like an editorial exclusion.
QWhat happens to rows a buyer already downloaded?
We cannot reach into a file already downloaded, so every removal is published: /api/datasets/removals lists each opted-out domain with the day it left, as a SHA-256 hash (the domain itself only to a key with dataset access). The dataset terms require a buyer to delete matching rows and to re-check at least every 30 days and before each re-use. Counts already published, such as totals and list sizes, are history and are not recomputed; named records are removed.
QDoes CrawlCheck pretend to be Googlebot or GPTBot?
Never to gain access. When a report measures what each agent receives, it deliberately sends those user-agent strings from its own address and records what comes back; a site that verifies crawler IPs should refuse them, and the report says that is a correct result.
QWhat is stored about my site?
A report at /r/<id> (kept until you delete it at /manage/data as the verified owner), one row per domain holding the measurements and scores so a re-scan can show what changed, and aggregate counters behind the dataset page. Nothing behind authentication is ever fetched.
QDoes the directory accept payment for placement or removal?
No. No vendor is listed for payment, no listing is removed for payment, and no score is edited on request. A dispute is a re-scan published as measured.

Keep reading