close
CrawlCheck
SDK · npm and PyPI

The agent SDK, on npm and on PyPI.

One package for JavaScript and one for Python, no dependencies in either. An agent gets a signed decision before it reads, cites, connects or transacts, uses a site’s own stapled verdict when one is served, and verifies every answer, receipt and log entry on its own machine.

1.9.1

@crawlcheck/sdk on npm

1.9.1

crawlcheck on PyPI

0

dependencies in either

129

API paths with generated types

npm i @crawlcheck/sdk
pip install crawlcheck
For agents

Three jobs it does for an agent

Each one runs the same way in both languages.

1 · Before acting

A signed decision

guard(domain, action) asks, verifies the receipt against the published keys and returns whether to proceed; preflight returns the receipt and leaves the call to you.

JavaScriptPython

2 · At the site

The site’s own staple

resolveFor uses the signed verdict a site serves in its CrawlCheck-Staple header with no lookup, and asks CrawlCheck only when the staple is missing, expired, for another domain or altered.

How it decidesThe answer format

3 · Trust nothing

Offline verification

verifyOffline checks a report’s whole evidence bundle down to its Bitcoin anchor; logCheck confirms an answer is in the witnessed log and catches a split view.

The checksProve the verifier

For site owners

One tag gives every visiting agent these tools

No account, no key, read-only.

Add this to any page and AI agents browsing your site (through WebMCP, or window.crawlcheckAgent) can ask CrawlCheck about your site before they act: whether to proceed, your real phone, address and hours, whether a value they hold is right, and which AI crawlers you have closed off. It sends only your hostname, never anything about the visitor.

<script src="https://crawlcheck.io/agent.js" async></script>

Tools: crawlcheck_check_this_site, crawlcheck_business_facts, crawlcheck_verify_fact, crawlcheck_ai_optout_record. Script sha256 6da6badcb890fa4f….

JavaScript

@crawlcheck/sdk 1.9.1

Node 20+ or any current browser.

From npm (npmjs.com/package/@crawlcheck/sdk), published from this repository by trusted publishing. The same tarball, byte for byte, is served here: npm i https://crawlcheck.io/sdk/crawlcheck-sdk-1.9.1.tgz

133176 bytes · sha256 14fadf528ddca4e333cebb7546c4ea20321bcf2a0034d26440194c81013bfdc2 · npm integrity sha512-gSP/iJ2DS5VcXn4bMBAlW7Hf8X9156PRyyL9KST4LxNwZrR+rF+JXXru0zu5LXWzhxPy+D2tFJI1+tYO3yKpyw== · verifier d628647818374e9e · types generated for 129 documented paths · versions served here: index.json (npm holds every release)

Check before acting

import { CrawlCheck } from "@crawlcheck/sdk";
const cc = new CrawlCheck();
const { proceed, receipt } = await cc.guard("example.com", "read", { agent: "MyAgentBot" });
// proceed is true on allow; warn proceeds only with { allowWarn: true }; require_confirmation only if confirm(receipt) resolves true.
// The receipt is a signed crawlcheck-decision; guard() verified it here and checked its key against the published directory.

Every verification returns integrity_valid, issuer_trusted and accepted. Act on accepted: it needs the signing key to be one CrawlCheck publishes, so a document someone signed with their own key is verified offline but never accepted. The client methods (verifyDocument, verifyOffline, verifyReceiptOffline, guard) check the published key directory for you.

preflight(domain, action, { agent, template, policy }) returns the signed receipt without deciding for you; verifyDocument(doc) verifies any signed resolve answer or decision receipt offline. resolve(domain) returns the full signed ResolveV1 answer.

Read a record

import { CrawlCheck } from "@crawlcheck/sdk";
const cc = new CrawlCheck();                       // { key: "cc_…" } for licence-gated fields
const mr = await cc.machineRecord({ domain: "example.com" });
mr.findings;            // gated exactly like the website: the top finding without a licence
mr.score_ledger;        // what each row and finding does to the grade

Every request and response type is generated from https://crawlcheck.io/openapi.json. Any documented route is callable with types checked against the contract:

await cc.get("/api/explain", { domain: "example.com", code: "NO_LLMS_TXT" });
await cc.get("/api/explain", { website: "x" });   // compile error: not a parameter of this operation

Use the site's staple

A site can serve its own signed verdict in the CrawlCheck-Staple header (and at /.well-known/crawlcheck-staple.json). resolveFor uses it when it verifies, with no request to CrawlCheck, and falls back to a signed lookup when it is missing, expired, for another domain or tampered:

const res = await fetch("https://shop.example.com/");
const v = await client.resolveFor("shop.example.com", { response: res });
v.source;              // "staple" or "network"
v.decisions.transact;  // allow | warn | require_confirmation | block | unsupported

stapleCheck(record, { host, publishedKids }) does the offline check on its own. Spec: https://crawlcheck.io/spec/staple

Do not trust the API: check the evidence yourself

const v = await cc.verifyOffline(mr.subject.report_id);   // downloads the bundle, verifies it HERE
v.checks;   // manifest hash, Ed25519 signature, key id, record digest, Merkle path to the day's root,
            // OpenTimestamps proof (Bitcoin), sealed roots over fetches, decisions and facts

ok: true passed, ok: false failed, ok: null could not run and why says why. A null is never a pass.

Remediation receipts (before, declared fix, deployment, after, verdict, signed):

import { verifyReceipt } from "@crawlcheck/sdk";
const rc = await cc.receipt("rc1:…");
const r = await verifyReceipt(rc);                                  // hash, key id, signature, order
const ours = (await cc.publishedKeyIds()).includes(rc.signature.kid);  // the one check offline cannot do
const accept = r.verified && ours;

Prove the verifier

const suite = await cc.conformance();   // 19 real bundles and receipts, valid and tampered, with expected outputs
suite.ok;                              // every fixture gave exactly its expected checks and verdict

npm test in the installed package runs the conformance suite and live checks against crawlcheck.io.

Catch a split view

Every Resolve answer names its leaf in CrawlCheck's transparency log and, once merged, carries the signed tree head of its batch. The client keeps every head it sees in client.log (plain JSON) and checks each new one against its neighbours: two heads for one batch, or a head that does not chain to the one before, is a **fork**, kept with both signed heads as proof. guard() refuses to proceed while the client holds one.

import fs from "node:fs";
import { CrawlCheck, newLogStore } from "@crawlcheck/sdk";
const store = fs.existsSync("cc-log.json") ? JSON.parse(fs.readFileSync("cc-log.json", "utf8")) : newLogStore();
const client = new CrawlCheck({ logStore: store });
await client.resolve("example.com");                 // the answer's head is checked against the store
const r = await client.logCheck();                   // current head + GitHub's witness co-signature + the chain down to your heads
console.log(r.summary);                              // "consistent: ..." or "SPLIT VIEW: ..."
fs.writeFileSync("cc-log.json", JSON.stringify(client.log));

logCheck verifies the witness we run on GitHub: a CI run re-checks the chain and co-signs the newest head with a GitHub OIDC token (RS256, GitHub's key) whose audience is the head's digest. A head the witness's chain does not contain shows up as a fork. Heads newer than the witnessed one are listed as unwitnessed. Clients can also swap heads directly: logHeads(store) on one side, client.logGossip(heads) on the other. Forks are reported to /api/v1/log/gossip (pass { report: false } to skip).

Errors

A refused request throws CrawlCheckError with status, body and locked (true for "this needs a licence for the domain").

Requires Node 20+ or any current browser (WebCrypto Ed25519).

Python · crawlcheck 1.9.1

The same calls from Python 3.8+, with no dependencies: resolve and batch resolve, the preflight policy engine and guard, resolve_for and staple_check for a site’s stapled verdict, log_check and split-view detection, the MCP tool lockfile, outcomes, and offline verification of resolve answers and decision receipts (pure-Python Ed25519, canonical JSON identical to the server’s).

pip install crawlcheck

From PyPI (pypi.org/project/crawlcheck), published from this repository by trusted publishing. The same wheel is served here: pip install https://crawlcheck.io/sdk/crawlcheck-1.9.1-py3-none-any.whl

from crawlcheck import CrawlCheck

cc = CrawlCheck()
receipt = cc.preflight("example.com", action="cite", agent="GPTBot", template="safe_citation")
print(receipt["decision"])            # allow | warn | require_confirmation | block | unsupported
assert cc.verify(receipt)["verified"]  # checked locally against the published key

ok, receipt = cc.guard("shop.example", action="transact", confirm=lambda r: ask_a_human(r))

Command line: python -m crawlcheck preflight example.com cite GPTBot · python -m crawlcheck verify receipt.json

38864 bytes · sha256 aa257783da118ed4d853017b9337d913591af2c2af9f86c486c91494012b6662 · built reproducibly by sdk-python/build_wheel.py