---
name: end-of-term-web-archive-ingestion
description: "How to get End of Term Web Archive datasets data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load the EOT 2024 URL-index inventory into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/end-of-term-web-archive
Provider: https://eotarchive.org/data/
Checked against the provider's documentation: 2026-11-01

## Access
- No account or key. The project publishes gzipped path manifests over HTTPS and S3.

## Request
```bash
curl -O 'https://eotarchive.s3.amazonaws.com/crawl-data/EOT-2024/eot-index.paths.gz'
```

## Format
A gzip-compressed newline-delimited manifest of URL-index object paths; prepend the public S3 HTTPS origin to form download URLs.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""End of Term 2024 URL-index inventory -> pandas -> Parquet. No key."""
import gzip
import requests
import pandas as pd

URL = "https://eotarchive.s3.amazonaws.com/crawl-data/EOT-2024/eot-index.paths.gz"
r = requests.get(URL, timeout=60)
r.raise_for_status()
paths = gzip.decompress(r.content).decode("utf-8").splitlines()
df = pd.DataFrame({"object_path": paths})
df["https_url"] = "https://eotarchive.s3.amazonaws.com/" + df["object_path"]
df.to_parquet("eot_2024_index_inventory.parquet", index=False)
print(len(df), "index objects", df["object_path"].iloc[0], "to", df["object_path"].iloc[-1])
```
Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 63 index objects eot-index/collections/EOT-2024/indexes/cluster.summary to eot-index/collections/EOT-2024/indexes/eot24-cdx-00061.cdxj.gz

## Limits
- Rate limit: Not stated.
- The 2024 dataset contains 1,216,891 WARC files totaling 2.29 PB compressed.
- The project says inventorying, staging and moving data into AWS is still ongoing.

## Quirks
- The manifest contains object paths, not archive records; prepend https://eotarchive.s3.amazonaws.com/ before retrieval.
- Use CDX or URL-index data to identify relevant captures before downloading large WARC objects.
- Crawler selection and crawl failures create gaps; a missing capture does not prove a live page did not exist.

## Terms and attribution
- The 2024 dataset page states that use, access and download are unrestricted.
- The project requests citation of the End of Term Web Archive project.
- Terms page: https://eotarchive.org/data/data-2024/

## Sources
- https://eotarchive.org/data/: "The End of Term project is working with the Amazon Web Services' Open Data Sponsorship Program to host a copy of the 2004, 2008, 2012, 2016, 2020 and 2024 End of Term Datasets."
- https://eotarchive.org/data/data-2024/: "The End of Term 2024 Dataset represents data collected by six collecting institutions from September 2024 through April 2025."
- https://eotarchive.org/data/data-2024/: "There are no restrictions on the use, access, and/or download of data from the End of Term Web Archive Dataset."
- https://eotarchive.org/data/data-2024/: "We request that you cite the End of Term Web Archive project when using the data provided from this dataset."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.