---
name: osti-api-ingestion
description: "How to get OSTI.GOV API (U.S. DOE research results) data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load OSTI.GOV records for a search into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/osti-api
Provider: https://www.osti.gov/api/v1/docs
Checked against the provider's documentation: 2026-11-03

## Access
- No key or account.
- Endpoint https://www.osti.gov/api/v1/records with q, field filters, rows and page; send Accept: application/json (or application/xml, application/x-bibtex).

## Request
```bash
curl -H 'Accept: application/json' 'https://www.osti.gov/api/v1/records?q=grid-scale+battery+storage&publication_date_start=01/01/2025&rows=100&page=1'
```

## Format
A JSON array of records with osti_id, title, product_type, publication_date, doi, authors, research_orgs, sponsor_orgs, description and links (citation and fulltext). X-Total-Count and Link headers carry paging.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""OSTI.GOV records (DOE-funded research) -> pandas -> Parquet. No key."""
import requests
import pandas as pd

params = {"q": "grid-scale battery storage", "publication_date_start": "01/01/2025", "rows": 100, "page": 1}
r = requests.get("https://www.osti.gov/api/v1/records", params=params,
                 headers={"Accept": "application/json"}, timeout=90)
r.raise_for_status()
recs = r.json()
rows = [{
    "osti_id": x.get("osti_id"),
    "title": x.get("title"),
    "product_type": x.get("product_type"),
    "publication_date": x.get("publication_date"),
    "doi": x.get("doi"),
    "research_orgs": "; ".join(x.get("research_orgs") or []),
    "url": next((l.get("href") for l in x.get("links", []) if l.get("rel") == "citation"), None),
} for x in recs]
df = pd.DataFrame(rows)
df.to_parquet("osti_records.parquet", index=False)
print(len(df), "records on this page; X-Total-Count:", r.headers.get("X-Total-Count"))
```
Run by us on 2026-11-03 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 100 records on this page; X-Total-Count: 1892

## Limits
- 20 records per page by default; set rows and page, and follow the Link header rel=next.
- A request rate limit is not stated.

## Quirks
- The v1 API is a release candidate and may change before the official v1 release.
- Date filters use MM/DD/YYYY, for example publication_date_start=01/01/2025.

## Terms and attribution
- Material may be read, downloaded and analyzed under the Government's reserved rights; doing more needs permission from the authors or publishers.
- OSTI asks AI/ML/LLM projects to contact apis@osti.gov first.
- Terms page: https://www.osti.gov/faqs

## Sources
- https://www.osti.gov/api/v1/docs: "The OSTI.GOV API allows you to query the Department of Energy's repository of research information resulting from DOE research funding."
- https://www.osti.gov/api/v1/docs: "Requests that return multiple records will be paginated to 20 records by default"
- https://www.osti.gov/faqs: "If OSTI.GOV users wish to do more than read, download, and analyze available material, they should contact the respective authors or publishers for permission."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.