arXiv API

Use in your agent

Load recent arXiv preprints in a category into pandas and Parquet. Checked against the provider's documentation on Nov 3, 2026.

Access

• No key or account.
• Endpoint https://export.arxiv.org/api/query with search_query, start, max_results, sortBy and sortOrder.

Request

curl 'https://export.arxiv.org/api/query?search_query=cat:q-fin.TR&start=0&max_results=100&sortBy=submittedDate&sortOrder=descending'

Format

Atom 1.0 XML feed: opensearch totalResults, startIndex and itemsPerPage, then one entry per e-print with id (abstract URL), title, summary, published, updated, author names, arxiv:primary_category, category terms and links (abstract page and PDF).

Load into pandas and Parquet

Needs Python 3 with requests, pandas and pyarrow.

"""arXiv API (Atom feed) -> pandas -> Parquet. No key; at most one request every three seconds."""
import xml.etree.ElementTree as ET
import requests
import pandas as pd

NS = {"a": "http://www.w3.org/2005/Atom", "arxiv": "http://arxiv.org/schemas/atom"}
params = {"search_query": "cat:q-fin.TR", "start": 0, "max_results": 100,
          "sortBy": "submittedDate", "sortOrder": "descending"}
r = requests.get("https://export.arxiv.org/api/query", params=params, timeout=60)
r.raise_for_status()
root = ET.fromstring(r.content)
rows = []
for e in root.findall("a:entry", NS):
    rows.append({
        "arxiv_id": e.findtext("a:id", namespaces=NS).rsplit("/abs/", 1)[-1],
        "title": " ".join(e.findtext("a:title", namespaces=NS).split()),
        "published": e.findtext("a:published", namespaces=NS),
        "primary_category": e.find("arxiv:primary_category", NS).get("term"),
        "authors": "; ".join(a.findtext("a:name", namespaces=NS) for a in e.findall("a:author", NS)),
        "abs_url": e.findtext("a:id", namespaces=NS),
    })
df = pd.DataFrame(rows)
df.to_parquet("arxiv_qfin_tr.parquet", index=False)
total = root.findtext("{http://a9.com/-/spec/opensearch/1.1/}totalResults")
print(len(df), "q-fin.TR entries of", total, "; newest:", df["published"].max())

Run by us on 2026-11-03 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 100 q-fin.TR entries of 2453 ; newest: 2026-11-01T11:08:08Z

Limits

• No more than one request every three seconds, one connection at a time, across all your machines.
• At most 2,000 results per call and 30,000 per query; larger requests return HTTP 400.

Quirks

• Titles and abstracts contain line breaks and repeated spaces; normalise whitespace.
• The entry id is the abstract URL; the arXiv identifier is the part after /abs/ and includes the version (for example v1).

Terms and attribution

• Descriptive metadata is CC0; e-prints keep their authors' or publishers' copyright and should not be stored and served from your servers without permission or a permitting licence.
• Direct users to arXiv.org for e-print content, and do not imply arXiv's endorsement.

Sources

• https://info.arxiv.org/help/api/tou.html: “When using the legacy APIs (including OAI-PMH, RSS, and the arXiv API), make no more than one request every three seconds, and limit requests to a single connection at a time.”
• https://info.arxiv.org/help/api/tou.html: “Direct users to arXiv.org to retrieve e-print content (PDF, source files, etc). We encourage you to link to the abstract page.”
• https://info.arxiv.org/help/api/index.html: “Acknowledge arXiv data usage with this statement on your product: “Thank you to arXiv for use of its open access interoperability.””