Ask before your agent acts.
Before an agent reads, cites, connects to or transacts with a domain, one call returns a signed decision: what the domain claims, what it actually serves, what CrawlCheck’s published rules decide for that action, and the evidence behind it. Pick your stack. Every block below is taken from this page and run in CI every day and after each SDK release, against a signing test server and the live MCP endpoint, so a block that stops working fails the build. Replace example.com and shop.example.com with the site your agent is about to use.
1
call before each action
0
keys or sign-up to ask
16
stacks below
Ed25519
signed, verifiable offline
From install to a verified receipt
Pick your stack. Each path ends with the same receipt, checked by the published verifier. The blocks are run from the published packages against this service every day and after each SDK release.
TypeScript / JavaScript
1 · run
npm i @crawlcheck/sdk2 · save as quickstart.mjs
import { writeFileSync } from "node:fs";
import { CrawlCheck } from "@crawlcheck/sdk";
const cc = new CrawlCheck();
const receipt = await cc.preflight("example.com", "read", { agent: "MyAgentBot/1.0" });
console.log("decision:", receipt.decision);
writeFileSync("receipt.json", JSON.stringify(receipt, null, 2));
const v = await cc.verifyDocument(receipt); // signature + the published key directory
console.log("accepted:", v.accepted);3 · run
node quickstart.mjs4 · verify (online: fetches the key directory)
curl -O https://crawlcheck.io/crawlcheck-verify.mjs
node crawlcheck-verify.mjs --online receipt.jsonThe last line printed is ACCEPTED: the receipt is authentic, signed by a key CrawlCheck publishes, for example.com and the action read. Authentic is not permission: act on the decision inside it. A block, warn or require_confirmation receipt is accepted just the same. Offline: save the key directory and the rotation record once (curl -o keys.json https://crawlcheck.io/.well-known/http-message-signatures-directory, curl -o rotation.json https://crawlcheck.io/.well-known/security-key-rotation.json) and run node crawlcheck-verify.mjs --keys keys.json --rotation rotation.json receipt.json; nothing is fetched, so a key retired or compromised after you saved them is not known until you refresh them.
Python
1 · run
pip install crawlcheck2 · save as quickstart.py
import json
from crawlcheck import CrawlCheck
cc = CrawlCheck()
receipt = cc.preflight("example.com", action="read", agent="MyAgentBot/1.0")
print("decision:", receipt["decision"])
json.dump(receipt, open("receipt.json", "w"), indent=2)
v = cc.verify(receipt) # signature + the published key directory
print("accepted:", v["accepted"])3 · run
python3 quickstart.py4 · verify (online: fetches the key directory)
curl -O https://crawlcheck.io/crawlcheck-verify.mjs
node crawlcheck-verify.mjs --online receipt.jsonThe last line printed is ACCEPTED: the receipt is authentic, signed by a key CrawlCheck publishes, for example.com and the action read. Authentic is not permission: act on the decision inside it. A block, warn or require_confirmation receipt is accepted just the same. Offline: save the key directory and the rotation record once (curl -o keys.json https://crawlcheck.io/.well-known/http-message-signatures-directory, curl -o rotation.json https://crawlcheck.io/.well-known/security-key-rotation.json) and run node crawlcheck-verify.mjs --keys keys.json --rotation rotation.json receipt.json; nothing is fetched, so a key retired or compromised after you saved them is not known until you refresh them.
MCP
1 · run
claude mcp add --transport http crawlcheck https://crawlcheck.io/mcp2 · run
curl -s https://crawlcheck.io/mcp \
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"preflight","arguments":{"domain":"example.com","action":"read"}}}' \
| python3 -c "import json,sys; print(json.load(sys.stdin)['result']['content'][0]['text'])" > receipt.json3 · verify (online: fetches the key directory)
curl -O https://crawlcheck.io/crawlcheck-verify.mjs
node crawlcheck-verify.mjs --online receipt.jsonThe last line printed is ACCEPTED: the receipt is authentic, signed by a key CrawlCheck publishes, for example.com and the action read. Authentic is not permission: act on the decision inside it. A block, warn or require_confirmation receipt is accepted just the same. Offline: save the key directory and the rotation record once (curl -o keys.json https://crawlcheck.io/.well-known/http-message-signatures-directory, curl -o rotation.json https://crawlcheck.io/.well-known/security-key-rotation.json) and run node crawlcheck-verify.mjs --keys keys.json --rotation rotation.json receipt.json; nothing is fetched, so a key retired or compromised after you saved them is not known until you refresh them.
Five answers, one rule
A policy you send can make a decision stricter, never looser. CrawlCheck recommends; your deployment’s own policy decides whether the agent acts, and every receipt names the policy that produced it.
| Decision | What your agent does |
|---|---|
| allow | proceed |
| warn | proceed only if your policy accepts the stated doubt |
| require_confirmation | a human approves first |
| block | do not act |
| unsupported | nothing to act on |
Templates: safe_citation, safe_data_retrieval, safe_api_connect, safe_mcp_tool, safe_commerce, safe_oauth, support_escalation. The tables behind every decision are at /rules/preflight.json.
Pick one and paste it
Start hereClaude Code, Claude Desktop or any MCP clientMCP clientAny languagePythonOpenAI Agents SDKClaude API tool useLangChain / LangGraphCrewAIVercel AI SDKBrowser agentsPlaywrightCursor, VS Code and other MCP clientsCloudflare Worker or any fetch() runtimeTypeScript / JavaScript SDKAt the site
Start here: enforce it where the action runs
Giving the model a preflight tool lets it ask; it does not make it ask. Put the check inside the function that performs the action, so a page, a tool result or a wrong plan cannot route around it. The SDK asks before the first request and again before every redirect to a new host, verifies each receipt against the published key and the exact domain, action and request, and proceeds only on allow (warn only if you opt in, require_confirmation only if your confirm() says yes, block, unsupported and every failure never). The outcome is reported back.
# pip install crawlcheck (1.1.0 or later)
from urllib.parse import urlparse
from crawlcheck import CrawlCheck
cc = CrawlCheck()
AGENT = "MyAgentBot/1.0 (+https://example.com/bot)"
class Refused(Exception):
pass
def ask_human(receipt):
return input("Allow " + receipt["action"] + " on " + receipt["domain"] + "? [y/N] ").strip().lower() == "y"
def fetch_page(url, action="read", allow_warn=True):
"""Every page fetch goes through here, so the model never decides whether the check runs. CrawlCheck is asked
before the first request and again before each redirect to a new host; a refused host is never contacted."""
r = cc.guarded_fetch(url, action=action, allow_warn=allow_warn, confirm=ask_human, headers={"user-agent": AGENT})
if not r["proceed"]:
raise Refused("; ".join(r["why"]))
cc.outcome(urlparse(r["url"]).hostname, action, status=r["status"])
return r["body"].decode("utf-8", "replace")
def guarded(action, method="POST"):
"""For an action that is not a page fetch (a form post, an order). Two separate checks: CrawlCheck decides whether
this destination is safe for the action, with the receipt bound to this exact method and URL; then a person
approves the exact order. Allowing a host never approves an order by itself."""
def wrap(fn):
def run(url, details, *args, **kwargs):
host = urlparse(url).hostname
ok, _ = cc.guard(host, action=action, operation={"method": method, "url": url}, confirm=ask_human)
if not ok:
raise Refused(host + ": " + "; ".join(cc.last_guard["why"]))
if input("Send this to " + host + "? " + repr(details) + " [y/N] ").strip().lower() != "y":
raise Refused("the person did not approve this exact order")
return fn(url, details, *args, **kwargs)
return run
return wrap
# warn = a stated doubt (say, two crawler identities refused). Accepting it for public reads is a policy choice
# (allow_warn=True above); guarded() keeps the default, refuse on warn, for connect and transact.
@guarded("transact")
def place_order(url, order):
... # your checkout call
Claude Code, Claude Desktop or any MCP client
Add the server once. The tools preflight (a signed decision for an action), resolve_domain (the full signed answer) and mcp_servers (what a domain's MCP endpoints actually answered) are then available to the model.
claude mcp add --transport http crawlcheck https://crawlcheck.io/mcp
# or in an MCP client config file:
{ "mcpServers": { "crawlcheck": { "type": "http", "url": "https://crawlcheck.io/mcp" } } }MCP client: refuse a server whose tools changed (Python)
Approve a server's tools once as a signed lockfile, then every connect is decided before any transport opens, the listed tools are compared with the lockfile, and a tool outside it is refused before tools/call. Nothing here calls a tool or reads arguments.
import asyncio, json
from mcp import ClientSession
from mcp.client.streamable_http import streamable_http_client
from crawlcheck import CrawlCheck
from crawlcheck.mcp import McpPreflight, McpPreflightRefused, with_preflight
cc = CrawlCheck()
url = "https://mcp.example.com/mcp"
# once, when you approve the server: keep this file with your config
# json.dump(cc.mcp_lock(url), open("mcp.lock.json", "w"))
lock = json.load(open("mcp.lock.json"))
async def main():
pf = McpPreflight(cc, lock=lock, confirm_destructive=True, confirm=lambda r: False)
await pf.before_connect(url) # raises McpPreflightRefused on block, unsupported or unverifiable
async with streamable_http_client(url) as (read, write, *_):
async with with_preflight(ClientSession(read, write), cc, server_url=url, preflight=pf) as s:
await s.initialize() # refused here if the tool list no longer matches the lockfile
await s.call_tool("search", {"q": "x"}) # refused if the tool is outside the lockfile
asyncio.run(main())
Any language: one HTTP call
Ask before the action. The answer is signed, so keep it with the action it authorised. From a client that can only fetch a URL, the same decision is GET https://crawlcheck.io/api/v1/preflight?domain=example.com&action=cite (agent and template are optional query parameters; a policy needs POST).
curl -s -X POST https://crawlcheck.io/api/v1/preflight \
-H 'content-type: application/json' \
-d '{"domain":"example.com","action":"cite","agent":"GPTBot","template":"safe_citation"}'Python
No dependencies. guard() asks, verifies the receipt locally against the published key, and only returns proceed=True when the decision allows it.
# pip install crawlcheck
from crawlcheck import CrawlCheck
cc = CrawlCheck()
proceed, receipt = cc.guard("example.com", action="read", agent="MyAgentBot")
if not proceed:
print("refused:", cc.last_guard["why"])
OpenAI Agents SDK (Python)
The tool the agent calls is the guarded one from the first block, so the check runs on every call whether or not the model thinks to ask. Or, with crawlcheck 1.11.0: Agent(..., tools=crawlcheck_tools(CrawlCheck())) from crawlcheck.openai_agents gives the agent all three CrawlCheck tools in one line: check a site before acting, verify a business fact, and fetch a page only after its site passes; each refuses when the check does.
from agents import Agent, Runner, function_tool
# fetch_page and Refused: the first block on this page
@function_tool
def read_page(url: str) -> str:
"""Fetch a web page. CrawlCheck is checked inside this tool."""
try:
return fetch_page(url)[:20000]
except Refused as e:
return "Not fetched: " + str(e)
agent = Agent(name="researcher", tools=[read_page], instructions="Use read_page to read websites.")
print(Runner.run_sync(agent, "Summarise https://example.com").final_output)Claude API tool use (Python)
Declare the action, not the check: when Claude asks to read a page, your code runs the guarded fetch.
import anthropic
# fetch_page and Refused: the first block on this page
client = anthropic.Anthropic()
tools = [{"name": "read_page", "description": "Fetch a web page by URL.",
"input_schema": {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"]}}]
def run_tool(block):
try:
text = fetch_page(block.input["url"])[:20000]
except Refused as e:
text = "Not fetched: " + str(e)
return {"type": "tool_result", "tool_use_id": block.id, "content": text}LangChain / LangGraph
The same guarded fetch as a LangChain tool; it works in any agent or LangGraph node. Or: crawlcheck_tools(CrawlCheck()) from crawlcheck.langchain (crawlcheck.llama_index for LlamaIndex; langchainTools(cc, lc) in @crawlcheck/sdk for LangChain.js) gives the agent all three CrawlCheck tools in one line: check a site before acting, verify a business fact, and fetch a page only after its site passes; each refuses when the check does.
from langchain_core.tools import tool
# fetch_page and Refused: the first block on this page
@tool
def read_page(url: str) -> str:
"""Fetch a web page by URL."""
try:
return fetch_page(url)[:20000]
except Refused as e:
return "Not fetched: " + str(e)CrewAI
The same guarded fetch as a CrewAI tool: the crew's agents call it, and the check runs inside it on every call. Or, with crawlcheck 1.11.0: Agent(..., tools=crawlcheck_tools(CrawlCheck())) from crawlcheck.crewai gives the agent all three CrawlCheck tools in one line: check a site before acting, verify a business fact, and fetch a page only after its site passes; each refuses when the check does.
from crewai import Agent, Crew, Task
from crewai.tools import tool
# fetch_page and Refused: the first block on this page
@tool("read_page")
def read_page(url: str) -> str:
"""Fetch a web page by URL. CrawlCheck is checked inside this tool."""
try:
return fetch_page(url)[:20000]
except Refused as e:
return "Not fetched: " + str(e)
researcher = Agent(role="researcher", goal="Summarise websites you are allowed to read", backstory="You check before you read.", tools=[read_page])
task = Task(description="Summarise https://example.com", expected_output="Three sentences, or why it was not fetched.", agent=researcher)
print(Crew(agents=[researcher], tasks=[task]).kickoff())Vercel AI SDK (TypeScript)
The tool fetches the page itself and checks first, so the model cannot skip the check. guardedFetch() verifies each receipt and checks again before following a redirect to another host. Or, with @crawlcheck/sdk 1.10.0: import * as ai from "ai"; generateText({ tools: vercelTools(new CrawlCheck(), ai), ... }) gives the agent all three CrawlCheck tools in one line: check a site before acting, verify a business fact, and fetch a page only after its site passes; each refuses when the check does.
// npm i ai zod @crawlcheck/sdk
import { tool } from "ai";
import { z } from "zod";
import { CrawlCheck } from "@crawlcheck/sdk";
const cc = new CrawlCheck();
export const readPage = tool({
description: "Fetch a web page by URL.",
inputSchema: z.object({ url: z.string().url() }),
execute: async ({ url }) => {
// asks before the first request and before every redirect to a new host
const r = await cc.guardedFetch(url, { agent: "MyAgentBot", allowWarn: true, init: { headers: { "user-agent": "MyAgentBot" } } });
if (!r.proceed) return "Not fetched: " + r.why.join("; ");
return (await r.response.text()).slice(0, 20000);
},
});
Browser agents (Browser Use and similar)
Check the start URL before handing the task to the browser agent, and stop on block or ask the user on require_confirmation. For per-page enforcement, gate navigation as in the Playwright block.
from urllib.parse import urlparse
from crawlcheck import CrawlCheck
cc = CrawlCheck()
def safe_to_start(url: str, action: str = "transact") -> bool:
proceed, receipt = cc.guard(urlparse(url).hostname, action=action, agent="MyBrowserAgent",
confirm=lambda r: input("Allow " + r["domain"] + "? [y/N] ").lower() == "y")
if not proceed:
print("CrawlCheck refused:", cc.last_guard["why"])
return proceed
if safe_to_start("https://shop.example.com/"):
pass # run the browser agent's task herePlaywright: a document-navigation gate
Each page load (document request) goes through preflight for read, so a scripted or agent-driven browser never opens a page on a domain the decision blocks; the answer is kept only until its receipt expires. What it does not cover is stated in the code. Before a purchase or sign-in, ask for that action separately, as in the decorator example, and have a person approve the exact order.
import { chromium } from "playwright";
import { CrawlCheck } from "@crawlcheck/sdk";
// A document-navigation gate. It checks each page load for read; it does not check scripts, images,
// fetch/XHR calls, form submissions or downloads, and allowing a host here never approves a purchase or a sign-in.
const cc = new CrawlCheck(), seen = new Map(); // host -> { ok, until }: kept only until the receipt expires
const browser = await chromium.launch(), page = await browser.newPage();
await page.route("**/*", async (route) => {
if (route.request().resourceType() !== "document") return route.continue();
const host = new URL(route.request().url()).hostname, hit = seen.get(host);
if (!hit || Date.now() >= hit.until) {
const r = await cc.guard(host, "read", { agent: "MyAgentBot", allowWarn: true });
const exp = Date.parse((r.receipt && r.receipt.expires_at) || "");
seen.set(host, { ok: r.proceed, until: Number.isFinite(exp) ? exp : Date.now() + 60000 });
}
return seen.get(host).ok ? route.continue() : route.abort("blockedbyclient");
});
await page.goto("https://example.com/");
await browser.close();
Cursor, VS Code and other MCP clients
The same remote MCP server, in each client's own config file. It is listed in the official MCP Registry as io.github.emmanuelorta/crawlcheck.
# Cursor: .cursor/mcp.json
{ "mcpServers": { "crawlcheck": { "url": "https://crawlcheck.io/mcp" } } }
# VS Code: .vscode/mcp.json
{ "servers": { "crawlcheck": { "type": "http", "url": "https://crawlcheck.io/mcp" } } }Cloudflare Worker or any fetch() runtime
Gate an outbound fetch: guardedFetch() asks, verifies each receipt against the published key and the exact domain, action and request, follows redirects itself and asks again before each new host.
import { CrawlCheck } from "@crawlcheck/sdk";
const cc = new CrawlCheck();
// Asks before the first request and before every redirect to a new host; a refused host is never contacted.
export async function readPage(url, agent = "MyAgentBot") {
const r = await cc.guardedFetch(url, { agent, allowWarn: true, init: { headers: { "user-agent": agent } } });
if (!r.proceed) throw new Error("CrawlCheck: " + r.why.join("; "));
return r.response;
}
TypeScript / JavaScript SDK
No dependencies, Node 20+ or any current browser. guard() asks, verifies the receipt against the published key, and returns proceed only when the decision allows it.
// npm i @crawlcheck/sdk (1.1.0 or later)
import { CrawlCheck } from "@crawlcheck/sdk";
const cc = new CrawlCheck();
const g = await cc.guard("example.com", "read", { agent: "MyAgentBot" });
if (!g.proceed) console.log("refused:", g.why);
At the site: show your receipt, use the site's staple
Send the receipt's digest with the request so the site can check you asked first, and let a site that staples its own signed verdict answer for itself: when the staple verifies, no call to CrawlCheck is made.
import { CrawlCheck } from "@crawlcheck/sdk";
const cc = new CrawlCheck();
const g = await cc.guard("shop.example.com", "read", { agent: "MyAgentBot" });
if (g.proceed) {
const res = await fetch("https://shop.example.com/", { headers: { ...g.header, "user-agent": "MyAgentBot" } });
const v = await cc.resolveFor("shop.example.com", { response: res });
v.source; // "staple" when the site's own verdict verified, "network" otherwise
v.decisions.transact; // check again before a different action
}
// Python: from crawlcheck import receipt_header; send receipt_header(receipt), then cc.resolve_for(domain, headers=dict(response.headers))
What the answer rests on
Every decision names the Resolve answer it was made from by its SHA-256, and both are signed with the key published at /keys. Verify either one offline with crawlcheck-verify.mjs, the SDKs, or python -m crawlcheck verify receipt.json. Reference: POST /api/v1/preflight · the Resolve specification · all endpoints · SDKs.
