---
name: openstreetmap-planet-ingestion
description: "How to get OpenStreetMap Planet Files data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load an OSM minute diff into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/openstreetmap-planet
Provider: https://planet.openstreetmap.org/
Checked against the provider's documentation: 2026-11-01

## Access
- No account or key. Replication state and compressed minute diffs are public downloads.
- The minute-diff example is a small incremental slice; initial full-database replication still requires a planet snapshot.

## Request
```bash
curl -O 'https://planet.openstreetmap.org/replication/minute/007/354/232.osc.gz'
```

## Format
Gzip-compressed OSM change XML with create, modify and delete groups containing node, way and relation elements. Elements include IDs, versions, timestamps, changesets, optional coordinates and nested tags or members.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""Latest OpenStreetMap minute replication diff -> pandas -> Parquet."""
import gzip
import xml.etree.ElementTree as ET
import requests
import pandas as pd

BASE = "https://planet.openstreetmap.org/replication/minute"
HEADERS = {"User-Agent": "SignalLabCheck/1.0"}
state = requests.get(f"{BASE}/state.txt", headers=HEADERS, timeout=30)
state.raise_for_status()
sequence = int(next(line.split("=", 1)[1] for line in state.text.splitlines()
                    if line.startswith("sequenceNumber=")))
path = f"{sequence:09d}"
diff_url = f"{BASE}/{path[:3]}/{path[3:6]}/{path[6:]}.osc.gz"
r = requests.get(diff_url, headers=HEADERS, timeout=60)
r.raise_for_status()
root = ET.fromstring(gzip.decompress(r.content))
rows = []
for group in root:
    action = group.tag if group.tag in {"create", "modify", "delete"} else "unknown"
    for obj in group:
        rows.append({
            "action": action, "type": obj.tag, "id": int(obj.attrib["id"]),
            "version": int(obj.attrib.get("version", 0)),
            "timestamp": obj.attrib.get("timestamp"), "lat": obj.attrib.get("lat"),
            "lon": obj.attrib.get("lon"), "changeset": obj.attrib.get("changeset"),
            "tag_count": sum(child.tag == "tag" for child in obj)
        })
df = pd.DataFrame(rows)
df["timestamp"] = pd.to_datetime(df["timestamp"], utc=True)
df[["lat", "lon"]] = df[["lat", "lon"]].apply(pd.to_numeric, errors="coerce")
df.to_parquet("osm_minute_diff.parquet", index=False)
print(len(df), "changed objects", "sequence", sequence,
      df["timestamp"].min().isoformat(), "to", df["timestamp"].max().isoformat())
```
Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 3502 changed objects sequence 7354232 2026-11-01T17:09:11+00:00 to 2026-11-01T17:10:10+00:00

## Limits
- Download rate limit: Not stated.
- The checked full snapshot was 89 GB as PBF and 167 GB as compressed XML.
- Weekly snapshots, daily diffs and hourly or minutely diffs have different lag and storage costs.

## Quirks
- Apply replication sequences in order and persist the last committed sequence; skipping or replaying a diff can make a local database inconsistent.
- Ways reference node IDs and relations reference nodes, ways or other relations, so a flat object table is only a staging representation.
- Delete records and some non-node objects have no latitude or longitude; preserve action, type, ID and version.
- The full-history files include older versions and deleted objects; the latest planet snapshot represents current data.

## Terms and attribution
- Current OpenStreetMap planet data is licensed under ODbL 1.0.
- Public use requires OpenStreetMap attribution and notice of the Open Database License.
- ODbL share-alike conditions can apply to altered or derivative databases; older pre-12 September 2012 files use CC BY-SA 2.0.
- Terms page: https://www.openstreetmap.org/copyright

## Sources
- https://wiki.openstreetmap.org/wiki/Planet.osm: "Planet files are very large (tens of GBs) so please check the size before starting to download."
- https://wiki.openstreetmap.org/wiki/Planet.osm: "A daily diff is generally about 40-80 MB, but can easily grow to 300-400 MB (both compressed) in exceptional cases."
- https://www.openstreetmap.org/copyright: "You are free to copy, distribute, transmit and adapt our data,"
- https://osmfoundation.org/wiki/Licence/Attribution_Guidelines: "The historical forms of attribution “© OpenStreetMap contributors” or “© OpenStreetMap” are acceptable."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.