---
name: crossref-rest-api-ingestion
description: "How to get Crossref REST API data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load Crossref works for a search into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/crossref-rest-api
Provider: https://www.crossref.org/documentation/retrieve-metadata/rest-api/
Checked against the provider's documentation: 2026-11-03

## Access
- No sign-up or key. Put your email address in the mailto parameter (or the User-Agent) to use the polite pool.
- Base URL https://api.crossref.org; /works takes query, filter, rows, select and cursor parameters.

## Request
```bash
curl 'https://api.crossref.org/works?query.bibliographic=monetary+policy+transmission&filter=from-pub-date:2025-01-01,type:journal-article&rows=100&mailto=you@example.org'
```

## Format
JSON with status, message-type and message; message carries total-results and items, each item a work with DOI, title (a list), container-title, published date-parts, is-referenced-by-count, publisher, and optional funder, license and abstract fields.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""Crossref works metadata -> pandas -> Parquet. No sign-up; add mailto for the polite pool."""
import requests
import pandas as pd

params = {
    "query.bibliographic": "monetary policy transmission",
    "filter": "from-pub-date:2025-01-01,type:journal-article",
    "rows": 100,
    "select": "DOI,title,container-title,published,is-referenced-by-count,publisher",
    "mailto": "you@example.org",  # replace with your own contact address
}
r = requests.get("https://api.crossref.org/works", params=params, timeout=60)
r.raise_for_status()
msg = r.json()["message"]
rows = [{
    "doi": it["DOI"],
    "title": (it.get("title") or [None])[0],
    "journal": (it.get("container-title") or [None])[0],
    "published": "-".join(str(p) for p in (it.get("published", {}).get("date-parts") or [[None]])[0] if p is not None),
    "cited_by": it.get("is-referenced-by-count"),
    "publisher": it.get("publisher"),
} for it in msg["items"]]
df = pd.DataFrame(rows)
df.to_parquet("crossref_works.parquet", index=False)
print(len(df), "works on this page of", msg["total-results"], "matches")
```
Run by us on 2026-11-03 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 100 works on this page of 140686 matches

## Limits
- Crossref's table: public pool 5 with concurrency 1, polite pool 10 with concurrency 3; on 2026-11-03 the headers showed 1 and 3 per 1s, so read x-rate-limit-limit and x-rate-limit-interval.
- A 429 means slow down; a manual block returns 403, which is why Crossref asks for a mailto in every request.

## Quirks
- title and container-title are lists; take the first element.
- Dates come as date-parts arrays such as [[2025, 3, 14]], sometimes with only a year.
- Use cursor=* for deep paging instead of large offsets.

## Terms and attribution
- Crossref says almost none of the metadata is subject to copyright and it may be used for any purpose.
- Some abstracts in the metadata may be under publishers' or authors' copyright.
- Terms page: https://www.crossref.org/documentation/retrieve-metadata/rest-api/

## Sources
- https://www.crossref.org/documentation/retrieve-metadata/rest-api/: "No sign-up is required to use the REST API, and almost none of the metadata is subject to copyright, and you may use it for any purpose."
- https://www.crossref.org/documentation/retrieve-metadata/rest-api/access-and-authentication/: "Anyone can access the REST API, no signup or registration is required."
- https://www.crossref.org/documentation/retrieve-metadata/rest-api/access-and-authentication/: "If you exceed the rate limits, you will receive a 429 HTTP response."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.