---
name: proofpoint-et-open-ingestion
description: "How to get Proofpoint Emerging Threats Open data into a pandas DataFrame and Parquet: access, the request, the format, limits, quirks and terms."
---

# Parse ET Open phishing rules into pandas and Parquet

Source entry: https://signal-lab-jet.vercel.app/datasets/proofpoint-et-open
Provider: https://rules.emergingthreats.net/OPEN_download_instructions.html
Checked against the provider's documentation: 2026-11-01

## Access
- No account or key. Category rule files and complete engine-specific bundles are public downloads.
- Choose the directory matching the deployed Snort or Suricata engine version.

## Request
```bash
curl -O 'https://rules.emergingthreats.net/open/suricata-7.0.3/rules/emerging-phishing.rules'
```

## Format
Suricata rule text with comments and one signature per active line; common options include msg, classtype, sid, rev, references, flow constraints and content matches.

## Load into pandas and Parquet
Needs Python 3 with requests, pandas and pyarrow.
```python
"""ET Open Suricata phishing rules -> pandas -> Parquet. Do not request indicators in rules."""
import re
import requests
import pandas as pd

URL = "https://rules.emergingthreats.net/open/suricata-7.0.3/rules/emerging-phishing.rules"
r = requests.get(URL, timeout=120)
r.raise_for_status()
rows = []
for line in r.text.splitlines():
    if not line or line.startswith("#"):
        continue
    sid = re.search(r";\s*sid:(\d+);", line)
    msg = re.search(r';\s*msg:"([^"]*)";', line)
    rev = re.search(r";\s*rev:(\d+);", line)
    classtype = re.search(r";\s*classtype:([^;]+);", line)
    rows.append({
        "sid": int(sid.group(1)) if sid else None,
        "msg": msg.group(1) if msg else None,
        "rev": int(rev.group(1)) if rev else None,
        "classtype": classtype.group(1) if classtype else None,
        "rule": line,
    })
df = pd.DataFrame(rows)
df.to_parquet("et_open_phishing_rules.parquet", index=False)
print(len(df), "active phishing rules", "SID range",
      int(df["sid"].min()), "to", int(df["sid"].max()))
```
Run by us on 2026-11-01 (python 3.13.5 | pandas 3.1.0 | pyarrow 26.0.0 | requests 2.34.2). Result: 2951 active phishing rules SID range 2015907 to 2072585

## Limits
- Updates are normally once daily on weekdays, with occasional out-of-band releases.
- Do not download the ruleset more often than once per hour; check version.txt first.
- Excessive requests can receive HTTP 429 and a 30-minute cool-down.

## Quirks
- Select rules for the deployed engine version; syntax and supported features vary across Snort and Suricata releases.
- Commented signatures are disabled and are skipped by this parser; local policy may enable or disable additional categories.
- Rules can contain domains, URLs or other indicators. Do not visit or fetch them; download only the rule file.

## Terms and attribution
- The Open distribution mixes GPL-2.0 and BSD-3-Clause-style rules according to SID ranges stated in its LICENSE file.
- Redistribution must retain the notices, conditions and disclaimers required by the applicable rule licence.
- Terms page: https://rules.emergingthreats.net/open/suricata-7.0.3/rules/LICENSE

## Sources
- https://rules.emergingthreats.net/OPEN_download_instructions.html: "ET is updated once daily on weekdays, and occasionally with out of band updates."
- https://rules.emergingthreats.net/OPEN_download_instructions.html: "We do not recommend downloading the ruleset more frequently than once per hour"
- https://rules.emergingthreats.net/OPEN_download_instructions.html: "until a 30 minute cool down period has elapsed."
- https://rules.emergingthreats.net/open/suricata-7.0.3/rules/LICENSE: "Rules with sids 2000000 through 2799999 are from Emerging Threats and are covered under the BSD License"

## Help improve this skill
Missing a source, or see something to improve? Call the MCP tool suggest_source or feedback (or POST /api/suggest or /api/feedback). Reviewed ideas: https://signal-lab-jet.vercel.app/agent-suggestions

Information for educational purposes only. Free tiers, limits and licenses belong to the providers and may change; check the provider's own terms before using the data.