---
name: ons-api-ingestion
description: "How to get UK Office for National Statistics API data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load the ONS dataset catalogue into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/ons-api
Provider: https://developer.ons.gov.uk/
Checked against the provider's documentation: 2026-11-04

## Access
- No account or key. Catalogue and observation endpoints return JSON.
- Identify automation with a contactable user agent and obey Retry-After.

## Request
```bash
curl 'https://api.beta.ons.gov.uk/v1/datasets?limit=100'
```

## Format
Paginated JSON items with dataset IDs, titles, descriptions, keywords, contacts, links, release frequency and state metadata.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""ONS Beta API dataset catalogue -> pandas -> Parquet. No key."""
import requests
import pandas as pd

URL = "https://api.beta.ons.gov.uk/v1/datasets"
headers = {"User-Agent": "signal-lab-catalog-check/1.0 (+https://signal-lab-jet.vercel.app)"}
r = requests.get(URL, params={"limit": 100}, headers=headers, timeout=60)
r.raise_for_status()
payload = r.json()
df = pd.json_normalize(payload["items"], sep=".")
keep = [c for c in ["id", "title", "description", "release_frequency", "next_release", "state"] if c in df]
df[keep].to_parquet("ons_datasets.parquet", index=False)
print(len(df), "datasets on page; total_count", payload.get("total_count"), "offset", payload.get("offset"))
```
Run by us on 2026-11-04 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 100 datasets on page; total_count 338 offset 0

## Limits
- 120 requests per 10 seconds and 200 per minute; 15 per 10 seconds for high-demand assets.
- Respect 429 Retry-After.

## Quirks
- The API is Beta.
- Pin edition and version before querying observations.
- Dataset-specific third-party rights can apply.

## Terms and attribution
- Most ONS content is OGL v3.0.
- Give source attribution and check exemptions.
- Terms page: https://www.ons.gov.uk/help/terms-conditions

## Sources
- https://developer.ons.gov.uk/: "The API is open and unrestricted - no API keys are required, so you can start using it immediately."
- https://developer.ons.gov.uk/bots/: "120 requests per 10 seconds - all site and API assets"
- https://www.ons.gov.uk/help/terms-conditions: "Most content on this website is subject to Crown copyright protection and is published under the Open Government Licence (OGL)."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.