---
name: phishtank-ingestion
description: "How to get Cisco Talos PhishTank data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Load the PhishTank verified-online JSON feed into Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/phishtank
Provider: https://phishtank.org/developer_info.php
Checked against the provider's documentation: 2026-11-01

## Access
- Create a free account and application key for automated file downloads.
- Set PHISHTANK_APP_KEY and use a descriptive User-Agent that identifies your application.

## Request
```bash
curl -H 'User-Agent: phishtank/your-application' 'https://data.phishtank.com/data/$PHISHTANK_APP_KEY/online-valid.json.bz2' -o online-valid.json.bz2
```

## Format
Bzip2-compressed JSON array with phish ID, URL, detail URL, submission and verification timestamps, verified and online flags, target brand, and nested IP/network details.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""PhishTank verified-online JSON feed -> pandas -> Parquet. Do not request listed URLs."""
import bz2
import json
import os
import requests
import pandas as pd

APP_KEY = os.environ["PHISHTANK_APP_KEY"]
URL = f"https://data.phishtank.com/data/{APP_KEY}/online-valid.json.bz2"
HEADERS = {"User-Agent": "phishtank/signal-lab-ingestion"}
r = requests.get(URL, headers=HEADERS, timeout=120)
r.raise_for_status()
records = json.loads(bz2.decompress(r.content))
df = pd.json_normalize(records, max_level=0)
df["details"] = df["details"].map(json.dumps)
for column in ["submission_time", "verification_time"]:
    df[column] = pd.to_datetime(df[column], utc=True)
df.to_parquet("phishtank_verified_online.parquet", index=False)
print(len(df), "verified online phishes", df["verification_time"].max().isoformat())
```
Not run: it needs a free API key.

## Limits
- Files update hourly.
- Keyless access is limited to a few downloads per day; exact keyed limits are returned in response headers.
- API requests over the current limit return HTTP 509.

## Quirks
- The download contains only records marked verified and online, so it excludes unverified reports and entries no longer considered operational.
- details is a nested list of IP and network observations; serialize it before writing Parquet or normalize it into a second table.
- The feed contains dangerous destinations. Do not visit or fetch listed URLs or hosts; request only the feed file.

## Terms and attribution
- The FAQ permits API use for commercial and non-commercial purposes.
- Automated downloads should use a free application key, and a descriptive User-Agent is required.
- Terms page: https://phishtank.org/faq.php

## Sources
- https://phishtank.org/developer_info.php: "Available in multiple formats and updated hourly, these make it easy to have fast and up to date phishing detection built into your application."
- https://phishtank.org/developer_info.php: "Without this key, you will be limited to a few downloads per day."
- https://phishtank.org/faq.php: "Is it OK to use the API for both commercial and non-commercial uses?"
- https://phishtank.org/api_info.php: "Going over-limit will result in an HTTP 509 error."

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.