Agent Ingestion

Machine-readable catalog of free datasets (financial markets, global macro, energy, agriculture and food, money flows, transport and shipping, research and grey literature, weather, news & RSS, OSINT and cyber threat), tools, and models for AI agents and automation.

Available Files

catalog.json

Complete catalog with metadata: name, category, badge, auth, free-tier limits, rate limits, and URLs.

Download

https://signal-lab-jet.vercel.app/catalog.json

catalog.csv

Same data in CSV format for spreadsheet tools and data analysis.

Download

https://signal-lab-jet.vercel.app/catalog.csv

llms.txt

Plain-text list of all 322 entries with a short header: notice, schemaVersion and field meanings.

Download

https://signal-lab-jet.vercel.app/llms.txt

llms-full.txt

Plain-text version with per-resource ingestion guidance for LLM context.

Download

https://signal-lab-jet.vercel.app/llms-full.txt

⚠️ Important Notice

Information presented for educational purposes only. Free tiers, limits, and licenses belong to the providers and may change. Users must verify and follow each provider's terms. Signal Lab does not host or redistribute third-party data.

Copy-Paste Prompt for any AI agent or MCP client

Use this prompt with your AI assistant to automatically build data loaders:

Fetch https://signal-lab-jet.vercel.app/catalog.json and build loaders for the data sources I need. Filter by type, badge (Free, Freemium, Open Source, Unofficial) and category. Read each entry's license and licenseNote, and respect each provider's terms, rate limits, and authentication requirements as stated in the catalog.

Python Starter Code

Filter catalog by badge, category, or authentication requirements:

import requests
import pandas as pd

# Load Signal Lab catalog
response = requests.get("https://signal-lab-jet.vercel.app/catalog.json", timeout=30)
response.raise_for_status()
catalog = response.json()
print(f"schemaVersion {catalog['schemaVersion']}, {catalog['count']} entries, generated {catalog['generated']}")

# Datasets (type == "Dataset"); no badge filter, so Free, Freemium and Unofficial entries are all kept
datasets = [r for r in catalog["resources"] if r["type"] == "Dataset"]

# Filter by category
market_data = [r for r in datasets if r["category"] == "Market Data"]

# Display results; read license and licenseNote before using data in a product
columns = ["name", "badge", "auth", "freeTierLimits", "license", "licenseNote", "lastVerified", "providerUrl"]
df = pd.DataFrame(datasets).reindex(columns=columns)
print(df.to_string(max_colwidth=40))

# Always check and respect provider terms and rate limits before use
print(f"\nNotice: {catalog['notice']}")

Use Cases

AI Agent Integration

Fetch catalog.json to discover available data sources, then build loaders dynamically based on your agent's needs.

Data Pipeline Automation

Filter by free tier limits and authentication to identify sources suitable for automated ingestion.

Research & Prototyping

Quickly identify free and open-source tools and datasets to build MVPs without upfront costs.

Compliance Checking

Reference authentication and rate limit fields to ensure your implementation respects provider terms.

Catalog Fields

Field	Description
name	Resource display name
slug	URL-friendly identifier
category	Resource category (e.g., Market Data, Crypto, Backtesting)
type	Kind of entry. Values in the catalog: "Dataset", "Tool", "Model", "Recipe", "Starter Kit".
domain	Data domain. Values in the catalog: "OSINT", "Financial", "Cyber Threat", "Transport & Shipping", "Agriculture & Food", "Money Flows", "Energy", "Global Macro", "Research & Grey Literature", "News & RSS", "Weather". Tools, models, recipes and starter kits are "Financial".
badge	Values in the catalog: "Free", "Freemium", "Open Source", "Unofficial". The homepage counts Free and Open Source as "fully free or open source" and Freemium and Unofficial as "with a free tier or unofficial access".
auth	Authentication requirement. Values in the catalog: "none", "free API key", "free account and API token", "none or free API key", "free account and API key", "free account", "account", "none (v1); registration key for v2", "free account; API key issued from the dashboard", "free registration key for the API; none for bulk files", "none; User-Agent header required", "none for the 24h/48h/7d files; free MAP_KEY for the API", "requested free access token", "none for read API and S3", "free account + approved M2M access + application token", "free Auth-Key for URLhaus and ThreatFox; none for checked Feodo Tracker and SSLBL files", "none for DROP; account and bearer token for Intelligence API", "none for limited access; free application key for automation", "none for 10/day; free API key with business email for 50/week", "free API key with Google account and Cloud project", "free API key with CrowdSec account", "Cisco.com account, client ID and client secret for OAuth 2.0", "none for public metadata/YARA; invite-only account for non-public data", "free account and API key; minor unauthenticated quotas", "free account and Personal Access Token", "none for HTTPS data and indexes", "free account and download token", "account and free licence key", "none for bulk files; API key for hosted API", "free account and OAuth token", "approved appname; API key for publishing", "free account, subscription key and API token", "none; free API key for higher limits", "none for files; free account and API token", "none for catalogue search; free account for downloads", "free API token", "none for basic use; free API key raises the daily budget 10x", "none (add mailto for the polite pool)", "none (optional free API key for more than 3 requests a second)", "none (email in User-Agent or mailto for a higher limit)", "none for search (API keys are mainly for publishers)", "none for public records search (guest limits); free account and access token for higher limits, authenticated and deposit operations", "none (60 requests/hour); free account token for up to 7,200/hour", "none (optional free access token)", "free client registration + OpenID Connect", "free registration", "none; optional API key for higher limits", "none for public headline; account for detail/export", "none for OGC/SOAP/REST services; free registration form for downloads", "free registration for API", "none (email address required to place an AccessAIS order)".
freeTierLimits	Free tier usage limits (if available)
rateLimits	API rate limits (if available)
license	Shows where the terms live: "Provider terms" is the provider's own terms of service, "Custom" is a license or notice specific to that provider, "unknown" marks Signal Lab originals whose reuse terms are not yet published, other values name a license. It is not a permission to use the data; read licenseNote and the linked terms. Values in the catalog: "Custom", "Provider terms", "CC-BY-4.0", "unknown", "CC0-1.0", "OGL-UK-3.0", "Apache-2.0", "MIT", "ODbL-1.0", "CC-BY-NC-SA-4.0", "BSD-3-Clause", "CC-BY-NC-4.0", "CC-BY-SA-4.0", "Unlicense", "GPL-2.0-only", "CC-BY-NC-SA-3.0", "CC-BY-3.0-IGO", "NLOD-2.0", "GPL-3.0-or-later", "BSD-2-Clause".
licenseUrl	Link to the terms or license page recorded for the entry
licenseNote	Entry-specific wording about the terms. Read it together with license and the linked terms before using data in a product
tasks	Task tags for the entry; the /browse page filters on them
lastVerified	Date (YYYY-MM-DD) we last compared the entry with the provider's pages
coverage	What a dataset covers (instruments, sources, history, delays), as free text recorded for the entry. Only Dataset entries have it. Recorded on 270 of 322 entries.
attribution	The attribution wording or rule recorded for the entry, as the provider states it (for example a credit line to display). Missing means not recorded, not that none is required. Recorded on 149 of 322 entries.
evidenceUrl	One provider page recorded as evidence for the entry (for example a terms, pricing or license page). Missing means not recorded. Recorded on 9 of 322 entries.
evidenceUrls	A list of provider pages recorded as evidence for the entry; a note may cite a page not in the list. Missing means not recorded. Recorded on 308 of 322 entries.
providerUrl	Official provider website or documentation
sitePageUrl	Signal Lab page with full details
docsUrl	Documentation or getting-started URL