---
name: arxiv-api-ingestion
description: "How to get arXiv API data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load recent arXiv preprints in a category into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/arxiv-api
Provider: https://info.arxiv.org/help/api/index.html
Checked against the provider's documentation: 2026-11-03

## Access
- No key or account.
- Endpoint https://export.arxiv.org/api/query with search_query, start, max_results, sortBy and sortOrder.

## Request
```bash
curl 'https://export.arxiv.org/api/query?search_query=cat:q-fin.TR&start=0&max_results=100&sortBy=submittedDate&sortOrder=descending'
```

## Format
Atom 1.0 XML feed: opensearch totalResults, startIndex and itemsPerPage, then one entry per e-print with id (abstract URL), title, summary, published, updated, author names, arxiv:primary_category, category terms and links (abstract page and PDF).

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""arXiv API (Atom feed) -> pandas -> Parquet. No key; at most one request every three seconds."""
import xml.etree.ElementTree as ET
import requests
import pandas as pd

NS = {"a": "http://www.w3.org/2005/Atom", "arxiv": "http://arxiv.org/schemas/atom"}
params = {"search_query": "cat:q-fin.TR", "start": 0, "max_results": 100,
          "sortBy": "submittedDate", "sortOrder": "descending"}
r = requests.get("https://export.arxiv.org/api/query", params=params, timeout=60)
r.raise_for_status()
root = ET.fromstring(r.content)
rows = []
for e in root.findall("a:entry", NS):
    rows.append({
        "arxiv_id": e.findtext("a:id", namespaces=NS).rsplit("/abs/", 1)[-1],
        "title": " ".join(e.findtext("a:title", namespaces=NS).split()),
        "published": e.findtext("a:published", namespaces=NS),
        "primary_category": e.find("arxiv:primary_category", NS).get("term"),
        "authors": "; ".join(a.findtext("a:name", namespaces=NS) for a in e.findall("a:author", NS)),
        "abs_url": e.findtext("a:id", namespaces=NS),
    })
df = pd.DataFrame(rows)
df.to_parquet("arxiv_qfin_tr.parquet", index=False)
total = root.findtext("{http://a9.com/-/spec/opensearch/1.1/}totalResults")
print(len(df), "q-fin.TR entries of", total, "; newest:", df["published"].max())
```
Run by us on 2026-11-03 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 100 q-fin.TR entries of 2453 ; newest: 2026-11-01T11:08:08Z

## Limits
- No more than one request every three seconds, one connection at a time, across all your machines.
- At most 2,000 results per call and 30,000 per query; larger requests return HTTP 400.

## Quirks
- Titles and abstracts contain line breaks and repeated spaces; normalise whitespace.
- The entry id is the abstract URL; the arXiv identifier is the part after /abs/ and includes the version (for example v1).

## Terms and attribution
- Descriptive metadata is CC0; e-prints keep their authors' or publishers' copyright and should not be stored and served from your servers without permission or a permitting licence.
- Direct users to arXiv.org for e-print content, and do not imply arXiv's endorsement.
- Terms page: https://info.arxiv.org/help/api/tou.html

## Sources
- https://info.arxiv.org/help/api/tou.html: "When using the legacy APIs (including OAI-PMH, RSS, and the arXiv API), make no more than one request every three seconds, and limit requests to a single connection at a time."
- https://info.arxiv.org/help/api/tou.html: "Direct users to arXiv.org to retrieve e-print content (PDF, source files, etc). We encourage you to link to the abstract page."
- https://info.arxiv.org/help/api/index.html: "Acknowledge arXiv data usage with this statement on your product: “Thank you to arXiv for use of its open access interoperability.”"

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.