---
name: neso-open-data-api-ingestion
description: "How to get NESO Open Data Portal API data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load the NESO dataset catalog into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/neso-open-data-api
Provider: https://www.neso.energy/data-portal/api-guidance
Checked against the provider's documentation: 2026-11-01

## Access
- No account or key for public CKAN action endpoints.
- This example retrieves catalog metadata; follow each resource URL to ingest a selected dataset.

## Request
```bash
curl 'https://api.neso.energy/api/3/action/package_search?rows=100'
```

## Format
CKAN JSON response containing total match count and dataset records with identifiers, titles, notes, tags, timestamps, licence labels and arrays of downloadable resources.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""NESO public dataset catalog metadata -> pandas -> Parquet."""
import requests
import pandas as pd

URL = "https://api.neso.energy/api/3/action/package_search"
response = requests.get(URL, params={"rows": 100}, timeout=60)
response.raise_for_status()
payload = response.json()["result"]
rows = []
for item in payload["results"]:
    rows.append({
        "name": item["name"],
        "title": item["title"],
        "metadata_created": item["metadata_created"],
        "metadata_modified": item["metadata_modified"],
        "license_title": item.get("license_title"),
        "resource_count": len(item.get("resources", [])),
    })
df = pd.DataFrame(rows)
df["metadata_created"] = pd.to_datetime(df["metadata_created"], utc=True)
df["metadata_modified"] = pd.to_datetime(df["metadata_modified"], utc=True)
df.to_parquet("neso_portal_catalog.parquet", index=False)
print(len(df), "datasets returned of", payload["count"], "matches;",
      int(df["resource_count"].sum()), "resources in returned metadata")
```
Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 100 datasets returned of 131 matches; 1040 resources in returned metadata

## Limits
- CKAN API pacing: maximum one request per second is requested.
- Datastore API pacing: maximum two requests per minute is requested.
- package_search is paginated; use rows and start to traverse more than the returned page.

## Quirks
- CKAN calls datasets packages and data files resources; portal labels use different terms.
- A dataset can contain many resources with different formats, periods and update times.
- The NESO licence applies only when the dataset explicitly associates itself with that licence.

## Terms and attribution
- The NESO licence permits commercial and non-commercial reuse of associated datasets.
- Associated data must carry the statement Supported by National Energy SO Open Data.
- Terms page: https://www.neso.energy/data-portal/neso-open-licence

## Sources
- https://www.neso.energy/data-portal/api-guidance: "It is recommended to limit requests to a maximum one request per second"
- https://www.neso.energy/data-portal/neso-open-licence: "The Licensor grants you a worldwide, royalty-free, perpetual, non-exclusive licence to use the Information subject to the conditions below."
- https://www.neso.energy/data-portal/neso-open-licence: "Supported by National Energy SO Open Data"
- https://www.neso.energy/data-portal/neso-open-licence: "each dataset on the NESO data portal is licenced on an individual basis"

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.