End of Term Web Archive datasets

Use in your agent

Load the EOT 2024 URL-index inventory into pandas and Parquet. Checked against the provider's documentation on Nov 1, 2026.

Access

• No account or key. The project publishes gzipped path manifests over HTTPS and S3.

Request

curl -O 'https://eotarchive.s3.amazonaws.com/crawl-data/EOT-2024/eot-index.paths.gz'

Format

A gzip-compressed newline-delimited manifest of URL-index object paths; prepend the public S3 HTTPS origin to form download URLs.

Load into pandas and Parquet

Needs Python 3 with requests, pandas and pyarrow.

"""End of Term 2024 URL-index inventory -> pandas -> Parquet. No key."""
import gzip
import requests
import pandas as pd

URL = "https://eotarchive.s3.amazonaws.com/crawl-data/EOT-2024/eot-index.paths.gz"
r = requests.get(URL, timeout=60)
r.raise_for_status()
paths = gzip.decompress(r.content).decode("utf-8").splitlines()
df = pd.DataFrame({"object_path": paths})
df["https_url"] = "https://eotarchive.s3.amazonaws.com/" + df["object_path"]
df.to_parquet("eot_2024_index_inventory.parquet", index=False)
print(len(df), "index objects", df["object_path"].iloc[0], "to", df["object_path"].iloc[-1])

Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 63 index objects eot-index/collections/EOT-2024/indexes/cluster.summary to eot-index/collections/EOT-2024/indexes/eot24-cdx-00061.cdxj.gz

Limits

• Rate limit: Not stated.
• The 2024 dataset contains 1,216,891 WARC files totaling 2.29 PB compressed.
• The project says inventorying, staging and moving data into AWS is still ongoing.

Quirks

• The manifest contains object paths, not archive records; prepend https://eotarchive.s3.amazonaws.com/ before retrieval.
• Use CDX or URL-index data to identify relevant captures before downloading large WARC objects.
• Crawler selection and crawl failures create gaps; a missing capture does not prove a live page did not exist.

Terms and attribution

• The 2024 dataset page states that use, access and download are unrestricted.
• The project requests citation of the End of Term Web Archive project.

Sources

• https://eotarchive.org/data/: “The End of Term project is working with the Amazon Web Services' Open Data Sponsorship Program to host a copy of the 2004, 2008, 2012, 2016, 2020 and 2024 End of Term Datasets.”
• https://eotarchive.org/data/data-2024/: “The End of Term 2024 Dataset represents data collected by six collecting institutions from September 2024 through April 2025.”
• https://eotarchive.org/data/data-2024/: “There are no restrictions on the use, access, and/or download of data from the End of Term Web Archive Dataset.”
• https://eotarchive.org/data/data-2024/: “We request that you cite the End of Term Web Archive project when using the data provided from this dataset.”