---
name: urlscan-api-ingestion
description: "How to get urlscan.io API data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Search urlscan.io metadata into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/urlscan-api
Provider: https://docs.urlscan.io/pages/api-intro
Checked against the provider's documentation: 2026-11-01

## Access
- Create a free urlscan.io account and store the API key in URLSCAN_API_KEY.
- This example searches existing metadata for example.com; it does not submit a URL for scanning or visit any returned site.

## Request
```bash
curl -H "API-Key: $URLSCAN_API_KEY" --get --data-urlencode 'q=page.domain:example.com' --data-urlencode 'size=100' 'https://urlscan.io/api/v1/search/'
```

## Format
JSON search response with a results array; each result can include task time, URL, UUID and visibility, page domain and IP, scan statistics and nested verdict summaries.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""Search existing urlscan.io scan metadata -> pandas -> Parquet. Requires URLSCAN_API_KEY."""
import os
import requests
import pandas as pd

URL = "https://urlscan.io/api/v1/search/"
headers = {"API-Key": os.environ["URLSCAN_API_KEY"]}
params = {"q": "page.domain:example.com", "size": 100}
r = requests.get(URL, headers=headers, params=params, timeout=30)
r.raise_for_status()
results = r.json().get("results", [])
df = pd.json_normalize(results, sep=".")
if "task.time" in df.columns:
    df["task.time"] = pd.to_datetime(df["task.time"], utc=True, errors="coerce")
df.to_parquet("urlscan_example_com_search.parquet", index=False)
print(len(df), "scan records", "columns", len(df.columns))
```
Not run: it needs a free API key.

## Limits
- Free quotas are account- and action-specific and apply per minute, hour and day.
- Only HTTP 200 responses count against quota; HTTP 429 means the action limit was exceeded.
- Search results and scan artifacts may be deleted at any time.

## Quirks
- Search uses Elasticsearch query-string syntax and field names are case-sensitive; escape reserved characters.
- A submitted scan initially returns 404 while processing and can later return 410 if deleted.
- Keep result URLs as data. Do not request returned third-party URLs or hosts.

## Terms and attribution
- Free community use is personal and non-commercial; commercial use requires permission or commercial terms.
- Submit only URLs you have permission to scan, and expect submitted data to be shared with the community and partners.
- Respect the action-specific quotas and implement backoff.
- Terms page: https://urlscan.io/terms/

## Sources
- https://docs.urlscan.io/pages/api-intro: "Our APIs allow you to submit URLs for scanning and to retrieve the results once the scan has finished."
- https://docs.urlscan.io/pages/api-rate-limits: "There are separate limits per minute, per hour, and per day for each action."
- https://docs.urlscan.io/pages/result-api-reference: "The Result API allows retrieving the metadata generated by each scan."
- https://urlscan.io/terms/: "Commercial use of any part of our service requires express written permission."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.