---
name: osv-dev-ingestion
description: "How to get OSV Vulnerability Database data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Query an OSV package version into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/osv-dev
Provider: https://google.github.io/osv.dev/api/
Checked against the provider's documentation: 2026-11-01

## Access
- No account or key. POST a package ecosystem, name and version to the public v1/query endpoint.

## Request
```bash
curl -X POST -H 'Content-Type: application/json' -d '{"package":{"name":"jinja2","ecosystem":"PyPI"},"version":"3.1.4"}' 'https://api.osv.dev/v1/query'
```

## Format
JSON vulnerability objects with OSV ID, aliases, summary, details, dates, affected packages and ranges, references, severity, credits and database-specific fields.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""OSV vulnerabilities for PyPI jinja2 3.1.4 -> pandas -> Parquet. No key."""
import json
import requests
import pandas as pd

URL = "https://api.osv.dev/v1/query"
payload = {"package": {"name": "jinja2", "ecosystem": "PyPI"}, "version": "3.1.4"}
r = requests.post(URL, json=payload, timeout=60)
r.raise_for_status()
rows = [{"id": v["id"], "summary": v.get("summary"),
         "published": v.get("published"), "modified": v.get("modified"),
         "aliases": json.dumps(v.get("aliases", []))}
        for v in r.json().get("vulns", [])]
df = pd.DataFrame(rows)
df["published"] = pd.to_datetime(df["published"], format="mixed", utc=True)
df["modified"] = pd.to_datetime(df["modified"], format="mixed", utc=True)
df.to_parquet("osv_jinja2_3_1_4.parquet", index=False)
print(len(df), "vulnerabilities for PyPI jinja2 3.1.4",
      ",".join(sorted(df["id"].tolist())))
```
Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 6 vulnerabilities for PyPI jinja2 3.1.4 GHSA-cpwx-vrp4-4pq7,GHSA-gmj6-6f8f-6699,GHSA-q2x7-8rv6-6q7h,PYSEC-2026-1471,PYSEC-2026-1472,PYSEC-2026-1475

## Limits
- Rate limit: currently no limits are stated for the API.
- HTTP/1.1 responses are limited to 32 MiB; HTTP/2 has no stated response-size limit.
- v1/query paginates above 1,000 vulnerabilities or when processing exceeds 20 seconds.

## Quirks
- Package ecosystem and name matching are case-sensitive except for registry-specific normalization documented by OSV.
- Timestamps can contain different fractional-second precision; parse them as mixed ISO-8601 values.
- Aliases can cause several source records to describe one underlying issue; retain IDs and aliases for deduplication.
- Large queries can return only next_page_token; continue until the token disappears even if a page has no vulnerabilities.

## Terms and attribution
- The aggregate combines upstream databases under different licences; apply the licence of each source record.
- Some included sources use attribution, share-alike, non-commercial or no-derivatives terms, so the whole dump cannot be treated as uniformly permissive.
- Terms page: https://google.github.io/osv.dev/data/

## Sources
- https://google.github.io/osv.dev/api/: "Currently there are no limits on the API."
- https://google.github.io/osv.dev/api/: "The API has a response size limit of 32MiB when using HTTP/1.1."
- https://google.github.io/osv.dev/data/: "This bucket contains a zip file with all vulnerabilities across all ecosystems (including withdrawn records)"
- https://google.github.io/osv.dev/data/: "GitHub Advisory Database"

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.