---
name: usda-esmis-wasde-ingestion
description: "How to get USDA ESMIS WASDE releases data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load the latest WASDE XML cells into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/usda-esmis-wasde
Provider: https://esmis.nal.usda.gov/publication/world-agricultural-supply-and-demand-estimates
Checked against the provider's documentation: 2026-11-01

## Access
- No account or key is required for the read-only ESMIS API or the linked release files.
- The release endpoint identifies the latest XML file without hard-coding a release path.

## Request
```bash
curl 'https://esmis.nal.usda.gov/api/v1/release/findByPubId/1659?page=0'
```

## Format
Paginated JSON release metadata with file URLs. Current WASDE releases link PDF, text, XLS and hierarchical XML files whose Cell elements inherit report, commodity, geography, market-year, forecast-month, attribute and unit context.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""Latest ESMIS WASDE XML cells -> pandas -> Parquet. No key."""
import io
import xml.etree.ElementTree as ET
import requests
import pandas as pd

API = "https://esmis.nal.usda.gov/api/v1/release/findByPubId/1659?page=0"
headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) SignalLabCheck/1.0"}
meta = requests.get(API, headers=headers, timeout=60)
meta.raise_for_status()
release = meta.json()["results"][0]
xml_url = next(url for url in release["files"] if url.endswith(".xml"))
r = requests.get(xml_url, headers=headers, timeout=120)
r.raise_for_status()
root = ET.parse(io.BytesIO(r.content)).getroot()
rows = []
def walk(node, context, path):
    current = dict(context)
    current.update(node.attrib)
    here = f"{path}/{node.tag}"
    if node.tag == "Cell":
        rows.append({"release_datetime": release["release_datetime"], "path": here, **current})
    for child in node:
        walk(child, current, here)
walk(root, {}, "")
df = pd.DataFrame(rows)
df.to_parquet("usda_esmis_wasde_latest.parquet", index=False)
print(len(df), "WASDE XML cells", release["release_datetime"], len(df.columns), "columns")
```
Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 11551 WASDE XML cells 2026-10-09T12:00:00+0000 106 columns

## Limits
- Rate limit: Not stated.
- The ESMIS API returns 25 releases per page by default.
- Machine-readable file availability varies across the archive.

## Quirks
- The XML is a hierarchical rendering of many report tables, not one rectangular dataset; the sample preserves inherited context and XML paths.
- Attribute names are table-specific, producing sparse columns after flattening.
- Retain release_datetime because monthly releases revise prior estimates and forecasts.

## Terms and attribution
- NAL says most information on its website is public domain and may be distributed or copied.
- NAL requests a byline credit and notes that specifically labelled third-party material can have separate restrictions.
- Terms page: https://www.nal.usda.gov/web-policies-and-important-links

## Sources
- https://esmis.nal.usda.gov/sites/default/files/swagger/esmis_public_api.yaml: "This documentation is for the read-only API endpoints of the ESMIS website."
- https://esmis.nal.usda.gov/publication/world-agricultural-supply-and-demand-estimates: "The report is released monthly, and provides annual forecasts for supply and use of U.S. and world wheat, rice, coarse grains, oilseeds, and cotton."
- https://www.nal.usda.gov/web-policies-and-important-links: "Most information on the NAL website is in the public domain and can be freely distributed or copied."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.