Common Crawl
liveReferenceCommon Crawl's public web archive — find every time a URL was crawled since 2008 and read the exact page bytes that were captured.
Tools
commoncrawl_crawlsList the Common Crawl monthly crawl collections (crawl ids like "CC-MAIN-2026-34") with the date range each one covers. AUTHORITATIVE for "which Common Crawl snapshot covers <date>" — call this first
No parameters required.
Try it
commoncrawl_index_searchSearch one Common Crawl collection's CDX index for archived captures of a URL, host, or whole domain. PREFER OVER WEB SEARCH when the question is "what did this page look like in <month>", "did Common
No parameters required.
Try it
commoncrawl_fetch_recordRead one archived page back out of Common Crawl by byte range — pass the filename, offset and length from a commoncrawl_index_search result and get the WARC headers, the captured HTTP response headers
No parameters required.
Try it
Test with curl
The gateway speaks JSON-RPC 2.0 over HTTP POST. You can test any pack directly from the terminal.
curl -X POST https://gateway.pipeworx.io/commoncrawl/mcp \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'curl -X POST https://gateway.pipeworx.io/commoncrawl/mcp \
-H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"commoncrawl_crawls","arguments":{}}}'Use with the SDK
Install @pipeworx/sdk to call tools from any TypeScript/Node project.
import { Pipeworx } from '@pipeworx/sdk';
const px = new Pipeworx();
const result = await px.call("commoncrawl_crawls", {});// Or ask in plain English:
const answer = await px.ask("common crawl's public web archive — find every time a url was crawled since 2008 and read the exact page bytes that were captured");