`commoncrawl_fetch_record`

Pack: commoncrawl · Endpoint: https://gateway.pipeworx.io/commoncrawl/mcp

No MCP client? Call it directly: GET https://gateway.pipeworx.io/v1/tools/commoncrawl_fetch_record for the schema, then POST the same URL with its arguments for the data.

Read one archived page back out of Common Crawl by byte range — pass the filename, offset and length from a commoncrawl_index_search result and get the WARC headers, the captured HTTP response headers, and the page body as it was crawled. AUTHORITATIVE for “what did this page actually say when it was crawled”, including pages that have since changed or gone offline. The body is truncated to a byte cap you control. Keyless.

Parameters

Name	Type	Required	Description
`filename`	string	yes	WARC path from a capture row, e.g. “crawl-data/CC-MAIN-2025-05/segments/1736703361969.6/warc/CC-MAIN-20250112194358-20250112224358-00433.warc.gz”.
`offset`	number	yes	Byte offset of the record within that WARC file (the capture row’s “offset”).
`length`	number	yes	Compressed byte length of the record (the capture row’s “length”). Max 10485760 (10 MB).
`max_body_bytes`	number	no	Truncate the decoded page body to this many bytes before returning it. Default 20000, max 200000. WARC headers and HTTP headers are always returned in full.

Example call

Arguments

{
  "filename": "crawl-data/CC-MAIN-2025-05/segments/1736703361969.6/warc/CC-MAIN-20250112194358-20250112224358-00433.warc.gz",
  "offset": 38994004,
  "length": 1225
}

curl

curl -X POST https://gateway.pipeworx.io/commoncrawl/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"commoncrawl_fetch_record","arguments":{"filename":"crawl-data/CC-MAIN-2025-05/segments/1736703361969.6/warc/CC-MAIN-20250112194358-20250112224358-00433.warc.gz","offset":38994004,"length":1225}}}'

TypeScript (@pipeworx/sdk)

import { Pipeworx } from '@pipeworx/sdk';
const pipeworx = new Pipeworx();

const result = await pipeworx.call('commoncrawl_fetch_record', {
  "filename": "crawl-data/CC-MAIN-2025-05/segments/1736703361969.6/warc/CC-MAIN-20250112194358-20250112224358-00433.warc.gz",
  "offset": 38994004,
  "length": 1225
});

Connect

Add this to your MCP client config, or use one-click install buttons:

{
  "mcpServers": {
    "commoncrawl": {
      "url": "https://gateway.pipeworx.io/commoncrawl/mcp"
    }
  }
}

See Getting Started for client-specific install steps.

commoncrawl_fetch_record