---
name: ncbi-eutilities-pubmed-ingestion
description: "How to get NCBI E-utilities (PubMed) data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load PubMed search results via E-utilities into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/ncbi-eutilities-pubmed
Provider: https://www.ncbi.nlm.nih.gov/books/NBK25497/
Checked against the provider's documentation: 2026-11-03

## Access
- No key needed up to 3 requests a second; an NCBI API key (free) allows more and goes in the api_key parameter.
- Send requests to https://eutils.ncbi.nlm.nih.gov with tool and email parameters so NCBI can contact you.

## Request
```bash
curl 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=heat+wave+mortality&retmode=json&retmax=100&tool=my-loader&email=you@example.org'
```

## Format
ESearch JSON: esearchresult with count and idlist (PubMed IDs). ESummary JSON: result with uids and one object per PMID carrying title, fulljournalname, pubdate, authors and articleids (including doi where present). EFetch returns full PubMed XML.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""PubMed via NCBI E-utilities (ESearch + ESummary) -> pandas -> Parquet. No key needed; at most 3 requests a second."""
import time
import requests
import pandas as pd

BASE = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
COMMON = {"db": "pubmed", "retmode": "json", "tool": "my-loader", "email": "you@example.org"}  # use your own tool name and email
s = requests.get(f"{BASE}/esearch.fcgi", params={**COMMON, "term": "heat wave mortality", "retmax": 100,
                 "sort": "pub_date", "datetype": "pdat", "mindate": "2025", "maxdate": "2026"}, timeout=60)
s.raise_for_status()
res = s.json()["esearchresult"]
time.sleep(0.4)
m = requests.get(f"{BASE}/esummary.fcgi", params={**COMMON, "id": ",".join(res["idlist"])}, timeout=60)
m.raise_for_status()
summ = m.json()["result"]
rows = [{
    "pmid": uid,
    "title": summ[uid].get("title"),
    "journal": summ[uid].get("fulljournalname"),
    "pubdate": summ[uid].get("pubdate"),
    "doi": next((i["value"] for i in summ[uid].get("articleids", []) if i.get("idtype") == "doi"), None),
} for uid in summ["uids"]]
df = pd.DataFrame(rows)
df.to_parquet("pubmed_summaries.parquet", index=False)
print(len(df), "PubMed summaries of", res["count"], "matches")
```
Run by us on 2026-11-03 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 100 PubMed summaries of 106 matches

## Limits
- No more than 3 requests per second without a key, counted across all E-utilities from one IP address.
- Run series of more than 100 requests at weekends or between 9 pm and 5 am Eastern Time on weekdays.

## Quirks
- ESearch only returns IDs; fetch summaries or records in a second call (ESummary or EFetch).
- pubdate is free text such as 2026 Oct 15 or 2026 Spring; parse it with care.

## Terms and attribution
- NLM asks to be acknowledged as the source; some PubMed abstracts may be under copyright held by publishers or authors.
- NCBI's Disclaimer and Copyright notice must be evident to users of a service built on the data.
- Terms page: https://www.ncbi.nlm.nih.gov/home/about/policies/

## Sources
- https://www.ncbi.nlm.nih.gov/home/about/policies/: "Make no more than 3 requests every 1 second."
- https://www.ncbi.nlm.nih.gov/home/about/policies/: "NCBI's Disclaimer and Copyright notice must be evident to users of your service."
- https://pubmed.ncbi.nlm.nih.gov/download/: "PubMed data are available via our FTP servers and via the E-utilities API."
- https://ftp.ncbi.nlm.nih.gov/pubmed/baseline/README.txt: "NLM freely provides PubMed data. Please note some abstracts may be protected by copyright."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.