@pipeworx/commoncrawl

Connect: https://gateway.pipeworx.io/commoncrawl/mcp · Install: one-click buttons

No MCP client? Skip the connection: POST https://gateway.pipeworx.io/v1/tools/search_packs {"query":"..."} to find a tool below, GET /v1/tools/<name> for its schema, POST the same URL with arguments for the data — see For AI agents.

Tools: 3

Common Crawl’s public web archive — find every time a URL was crawled since 2008 and read back the exact page bytes that were captured.

Tools

• commoncrawl_crawls(limit?, contains_date?) — the monthly crawl collections (CC-MAIN-2026-34 and friends) with the capture window each one covers. contains_date answers “which snapshot would have seen my page on that day”.
• commoncrawl_index_search(url, crawl?, match_type?, limit?, page?, filter?, from?, to?) — CDX index lookup for a URL, host or whole domain. Returns capture timestamp, HTTP status, MIME type, detected language, content digest, and the WARC filename + offset + length that the next tool needs.
• commoncrawl_fetch_record(filename, offset, length, max_body_bytes?) — reads one archived record back by byte range: WARC headers, the captured HTTP response headers, and the page body as crawled.

Auth

Keyless.

Data sources

• https://index.commoncrawl.org/collinfo.json — the crawl collection list.
• https://index.commoncrawl.org/{crawl}-index — the CDX index, JSON-lines.
• <https://data.commoncrawl.org/{warc path}> — WARC records, HTTP range reads.

Things the next person would otherwise rediscover

• The CDX host 502s sporadically under load. Measured 2026-09-17: the same url=example.com query alternated between 200 and nginx 502 inside a minute, so it is load, not the query. ccFetch retries a 5xx twice with a short backoff. Without that a caller reads transient nginx noise as “Common Crawl has no record of this URL”, which is a different and wrong answer.
• “No captures” is a 404 with an English sentence, not JSON. Handled as an empty result set with an explicit note, so an empty captures array is never silently indistinguishable from an upstream failure.
• Each indexed record is its own gzip member, so a byte-range read of offset..offset+length-1 decompresses standalone — no need to stream the whole 1 GB WARC. DecompressionStream('gzip') handles it in the Workers runtime.
• new TextDecoder('utf-8', { fatal: false }) does not typecheck against @cloudflare/workers-types: its TextDecoderConstructorOptions requires ignoreBOM too. Pass no options; non-fatal is the default anyway.
• A broad match_type: "domain" query over a large domain is expensive upstream and is the first thing to 502. Narrow with match_type: "host" plus a filter such as =status:200.

Related packs

crawlgraph is our own link graph over sites we crawl. This pack reads the Common Crawl Foundation’s corpus — different data, different questions.

Tools

• commoncrawl_crawls — List the Common Crawl monthly crawl collections (crawl ids like “CC-MAIN-2026-34”) with the date range each one covers. AUTHORITATIVE for “which Common Crawl snapshot covers ” — call this first
• commoncrawl_index_search — Search one Common Crawl collection’s CDX index for archived captures of a URL, host, or whole domain. PREFER OVER WEB SEARCH when the question is “what did this page look like in ”, “did Common
• commoncrawl_fetch_record — Read one archived page back out of Common Crawl by byte range — pass the filename, offset and length from a commoncrawl_index_search result and get the WARC headers, the captured HTTP response headers