---
name: common-crawl-ingestion
description: "How to get Common Crawl web corpus and indexes data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load the Common Crawl collection catalog into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/common-crawl
Provider: https://commoncrawl.org/get-started
Checked against the provider's documentation: 2026-11-01

## Access
- No account or key for the collection catalog, CDX URL index or HTTPS data downloads.
- Use a descriptive User-Agent and slow down on CDX HTTP 503 responses.

## Request
```bash
curl 'https://index.commoncrawl.org/collinfo.json'
```

## Format
JSON array of crawl collections with id, name, timegate, cdx-api, from and to fields.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""Common Crawl crawl catalog -> pandas -> Parquet. No key."""
import requests
import pandas as pd

URL = "https://index.commoncrawl.org/collinfo.json"
headers = {"User-Agent": "SignalLabCheck/1.0 research@example.invalid"}
r = requests.get(URL, headers=headers, timeout=60)
r.raise_for_status()
df = pd.DataFrame(r.json())
df.to_parquet("common_crawl_collections.parquet", index=False)
print(len(df), "crawl indexes", df["id"].iloc[0], "to", df["id"].iloc[-1])
```
Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 129 crawl indexes CC-MAIN-2026-43 to CC-MAIN-2008-2009

## Limits
- Rate limit: Not stated.
- The CDX API is heavily rate limited; HTTP 503 means slow down, and a temporary IP block may last 24 hours.
- Use the URL Index through Athena or Spark for broad or high-volume filtering.

## Quirks
- collinfo.json is newest-first; do not assume the last element is the latest crawl.
- The crawl is a sample of the web and does not generally archive every page of a site.
- WARC contains raw responses, WAT contains derived metadata and WET contains extracted text; select the smallest format that fits the task.

## Terms and attribution
- Use of the service and crawled content is subject to Common Crawl's Terms of Use.
- Crawled content may be governed by source-site terms and third-party intellectual-property rights.
- The terms prohibit collecting personal information for use separately from the crawled content.
- Terms page: https://commoncrawl.org/terms-of-use

## Sources
- https://commoncrawl.org/get-started: "Crawl data is free to access by anyone from anywhere."
- https://commoncrawl.org/overview: "The Common Crawl corpus contains petabytes of data, regularly collected since 2008."
- https://index.commoncrawl.org/: "All data and index files are free to download."
- https://commoncrawl.org/faq: "Our CDX API endpoint is frequently abused and therefore heavily rate limited."
- https://commoncrawl.org/terms-of-use: "BY USING THE CRAWLED CONTENT, YOU AGREE TO RESPECT THE COPYRIGHTS AND OTHER APPLICABLE RIGHTS OF THIRD PARTIES IN AND TO THE MATERIAL CONTAINED THEREIN."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.