Next free-source huntWe check for new free data every weekday.

Methodology

How we check entries and score them

Every dataset entry is checked once a day. Entry pages show what the check measured under “Last checked”, and the measured score is built only from those measurements and the catalog's reviewed licence and auth fields.

// 01 / I// Latest run

Latest run

Run of 2026-11-21: 1,528 sources checked, 1,275 reachable, 44 not reachable, 65 blocking automated checks, 144 not requested because of robots.txt. Published Nov 21, 2026, 8:30 AM ET.

Python how-tos: 131 ran OK, 6 failed, 15 stopped at a limit, 53 need a key (not run), 31 not run (other reason). Checks began Nov 19, 2026, 1:39 PM ET.

Per-entry results: /api/entry-checks/<slug>. Run totals: /api/entry-checks/summary.

// 02 / II// The daily check

The daily check

The checks run on Signal Lab's own compute cluster through its job queue, once a day per entry. For each entry the check requests one URL the entry itself lists: the data endpoint when the entry needs no key, otherwise the provider's page. An endpoint that needs a key is never called.

  • One small request. A GET asking for the first 64 KB only (a ranged request); the check reads at most 64 KB and closes the connection. Bulk files are never downloaded.
  • robots.txt first. If the site's robots.txt does not allow it, nothing is requested and the entry shows “not requested”; that check counts neither as up nor down.
  • Blocks automated checks. When the site answers with HTTP 403 or 429, or with a bot challenge page (for example a Cloudflare “Just a moment” or captcha page), the entry shows “blocks automated checks” with the status it returned. The answer is recorded and shown, also when the same URL worked before, but it counts neither as up nor down: a block on automated requests says nothing about whether the source works for a person or a keyed client.
  • Identified. User-Agent SignalLabCheck/1.0 (+https://signal-lab.qdt.ai/methodology; signal-lab@qdt.ai). No cookies, keys or sign-in. Requests to the same host are spaced out.
  • What is recorded. The HTTP status, the time to the first response, the Last-Modified header, the newest date found in the part read, the format, and the field names seen. A few sample rows are shown only when the entry's licence allows redistribution and the screening check does not flag them; otherwise field names only.
  • Change detection. A short fingerprint of the part read is compared with the previous check; when it differs, the entry shows that the data changed between those two checks.
// 03 / III// Running the how-to

Running the how-to

When an entry has a Python how-to (its ingestion skill, or the snippet on its page) and needs no key, the check runs it as published, in an isolated sandbox: no secrets or environment variables, 60 seconds of CPU and 90 seconds in total, at most 25 connections (public addresses, web ports only) and 10 MB received. Because a how-to downloads real data, it runs at most once a week per entry; the page shows the date of the last run. It counts as OK when it produces a pandas DataFrame with rows.

A how-to that reads a key or token, or belongs to an entry that needs one, shows “needs a key, not run”. One that stops at the size or request limit shows that it stopped, which is not counted as a failure. Neither is a how-to that needs a Python package the sandbox does not have, or one whose host's robots.txt does not allow automated requests: those show “not run” and the reason.

// 04 / IV// Measured score (v2)

Measured score (v2)

score = round(100 × Σ wᵢ·cᵢ ÷ Σ wᵢ), over the measured components only

Each component cᵢ is between 0 and 1. A component that could not be measured for an entry is left out of both sums, and the entry page says which components were used.

ComponentWeightHow it is measured
Uptime35Share of the last 30 checks in which the checked URL answered with HTTP 2xx. Checks not requested because of robots.txt, and checks the site answered with a block on automated checks, are left out of both the count and the total. If every check was blocked or skipped, uptime is not measured.
Freshness20Age of the newest measured change: the server's Last-Modified header, the newest date in the part read, or the check at which what we read changed. Steps below.
Licence clarity15From the catalog's reviewed licence fields: 1 for a standard licence with a published text (SPDX id), 0.5 for the provider's own terms with a terms link and a reviewed note, 0.25 with only one of the two, 0 when unknown. It measures how clearly the terms are stated, not how open they are.
Auth friction15From the catalog's auth field: 1 no key or account, 0.75 none for part of the access, 0.5 a free key, token or account, 0.25 access on request or approval, 0 paid.
How-to ran151 when the entry's Python how-to produced a DataFrame with rows on the latest check, 0 when it failed. Not measured (left out) when there is no how-to, it needs a key, or it stopped at the check's limits.
Total100

Freshness steps (days since the newest measured change → value): ≤ 7 → 1, ≤ 31 → 0.8, ≤ 92 → 0.6, ≤ 366 → 0.4, ≤ 1096 → 0.2, older → 0. Archives that are no longer updated score low here by design; the other components still count.

Not enough checks yet. An entry needs at least 3 daily checks before it gets a number; until then its page says “not enough checks yet”. No score is estimated or carried over.

The score is new for dataset entries and is shown next to the existing fields; it does not replace anything yet. The formula is versioned: a change in weights or steps gets a new version number here.

// 05 / V// Limits

Limits

  • One request a day shows that a URL answered at that moment, not that every part of the service works.
  • Last-Modified is the server's own claim; the newest date in the data comes only from the first 64 KB.
  • A provider page answering does not prove its data downloads work; entries checked at their provider page say so.
  • HTTP 403, HTTP 429 and bot challenge pages are recorded as “blocks automated checks” and left out of uptime, like robots.txt skips. A site that blocks every check therefore has no uptime figure; its score uses the other measured components. Any other HTTP error (for example 404 or 500) and any network failure count as not reachable.
  • Licence clarity and auth friction come from the catalog's reviewed fields, not from the check. See an entry's licence note for the actual terms.

Questions or corrections: send feedback.