`commoncrawl_index_search`

Pack: commoncrawl · Endpoint: https://gateway.pipeworx.io/commoncrawl/mcp

No MCP client? Call it directly: GET https://gateway.pipeworx.io/v1/tools/commoncrawl_index_search for the schema, then POST the same URL with its arguments for the data.

Search one Common Crawl collection’s CDX index for archived captures of a URL, host, or whole domain. PREFER OVER WEB SEARCH when the question is “what did this page look like in ”, “did Common Crawl ever see this URL”, or “list the URLs crawled under this domain” — it returns the crawl record (timestamp, HTTP status, MIME type, detected language, content digest) plus the WARC filename/offset/length that commoncrawl_fetch_record needs to read the page bytes. Keyless.

Parameters

Name	Type	Required	Description
`url`	string	yes	URL, host, or domain to look up, e.g. “example.com”, “https://www.nasa.gov/news”, “*.python.org”.
`crawl`	string	no	Crawl collection id from commoncrawl_crawls, e.g. “CC-MAIN-2026-34”. Defaults to the most recent crawl.
`match_type`	string	no	How to match `url`: “exact” (that URL only), “prefix” (that path and everything under it), “host” (every URL on that exact host), “domain” (that host and all subdomains). Default “exact”.
`limit`	number	no	Maximum captures to return. Default 20, max 200.
`page`	number	no	Zero-based page number for paging through a large domain query. Default 0.
`filter`	string	no	Optional CDX field filter, e.g. “=status:200” (only successful captures), “=mime:text/html”, “~url:.blog.”. Repeatable filters are not supported here — pass one.
`from`	string	no	Optional lower bound on capture timestamp, as YYYY, YYYYMM, or YYYYMMDD, e.g. “202501”.
`to`	string	no	Optional upper bound on capture timestamp, same format as `from`.

Example call

Arguments

{
  "url": "example.com",
  "crawl": "CC-MAIN-2025-05",
  "limit": 3
}

curl

curl -X POST https://gateway.pipeworx.io/commoncrawl/mcp \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"commoncrawl_index_search","arguments":{"url":"example.com","crawl":"CC-MAIN-2025-05","limit":3}}}'

TypeScript (@pipeworx/sdk)

import { Pipeworx } from '@pipeworx/sdk';
const pipeworx = new Pipeworx();

const result = await pipeworx.call('commoncrawl_index_search', {
  "url": "example.com",
  "crawl": "CC-MAIN-2025-05",
  "limit": 3
});

More examples

{
  "url": "blog.python.org",
  "crawl": "CC-MAIN-2025-05",
  "match_type": "host",
  "filter": "=status:200",
  "limit": 5
}

Connect

Add this to your MCP client config, or use one-click install buttons:

{
  "mcpServers": {
    "commoncrawl": {
      "url": "https://gateway.pipeworx.io/commoncrawl/mcp"
    }
  }
}

See Getting Started for client-specific install steps.

commoncrawl_index_search