Common Crawl web corpus and indexes

Use in your agent

Load the Common Crawl collection catalog into pandas and Parquet. Checked against the provider's documentation on Nov 1, 2026.

Access

• No account or key for the collection catalog, CDX URL index or HTTPS data downloads.
• Use a descriptive User-Agent and slow down on CDX HTTP 503 responses.

Request

curl 'https://index.commoncrawl.org/collinfo.json'

Format

JSON array of crawl collections with id, name, timegate, cdx-api, from and to fields.

Load into pandas and Parquet

Needs Python 3 with requests, pandas and pyarrow.

"""Common Crawl crawl catalog -> pandas -> Parquet. No key."""
import requests
import pandas as pd

URL = "https://index.commoncrawl.org/collinfo.json"
headers = {"User-Agent": "SignalLabCheck/1.0 research@example.invalid"}
r = requests.get(URL, headers=headers, timeout=60)
r.raise_for_status()
df = pd.DataFrame(r.json())
df.to_parquet("common_crawl_collections.parquet", index=False)
print(len(df), "crawl indexes", df["id"].iloc[0], "to", df["id"].iloc[-1])

Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 129 crawl indexes CC-MAIN-2026-43 to CC-MAIN-2008-2009

Limits

• Rate limit: Not stated.
• The CDX API is heavily rate limited; HTTP 503 means slow down, and a temporary IP block may last 24 hours.
• Use the URL Index through Athena or Spark for broad or high-volume filtering.

Quirks

• collinfo.json is newest-first; do not assume the last element is the latest crawl.
• The crawl is a sample of the web and does not generally archive every page of a site.
• WARC contains raw responses, WAT contains derived metadata and WET contains extracted text; select the smallest format that fits the task.

Terms and attribution

• Use of the service and crawled content is subject to Common Crawl's Terms of Use.
• Crawled content may be governed by source-site terms and third-party intellectual-property rights.
• The terms prohibit collecting personal information for use separately from the crawled content.

Sources

• https://commoncrawl.org/get-started: “Crawl data is free to access by anyone from anywhere.”
• https://commoncrawl.org/overview: “The Common Crawl corpus contains petabytes of data, regularly collected since 2008.”
• https://index.commoncrawl.org/: “All data and index files are free to download.”
• https://commoncrawl.org/faq: “Our CDX API endpoint is frequently abused and therefore heavily rate limited.”
• https://commoncrawl.org/terms-of-use: “BY USING THE CRAWLED CONTENT, YOU AGREE TO RESPECT THE COPYRIGHTS AND OTHER APPLICABLE RIGHTS OF THIRD PARTIES IN AND TO THE MATERIAL CONTAINED THEREIN.”