// MCP servers · method

How we audit MCP servers.

An audit you can check: what we test, what we never do, and how we keep it fair.The code that runs the audit follows exactly what this page says.

Auditor version 1.0.0. See the results

The public audit

Once a week we connect to every listed server the way an AI client like Claude, ChatGPT or Cursor would. We complete the MCP handshake, list its tools, resources and prompts, and read everything the server says about itself.

We do not run its tools. A public audit is the view any client has before it calls anything: it shows whether a server works, whether it needs sign-up, whether its tools say they can change things, and whether its text tries to steer your AI. It does not prove that the data a server returns is correct: that is what the agent tests below are for.

What we check

Connection

It responds: We connect to the address in its registry listing. No answer within 15 seconds, after one retry, is "Not responding". A handshake slower than 5 seconds is a warning.
It is encrypted: The address must use HTTPS. Plain HTTP fails: anything sent to it can be read or changed on the way.

Sign-in

Whether it needs an account: If the server asks for sign-in, we stop there and say so: the tool checks need an account, so the result is a limited audit.
Whether sign-in is discoverable: MCP clients find out how to sign in from OAuth protected-resource metadata (RFC 9728). Without it, people must set up access by hand: a warning.

Protocol

The MCP handshake: We send the standard initialize request and record the protocol version the server picks. A handshake that fails, after one retry, is "Not working".
Protocol version and transport: We ask for the latest version (2025-11-25). Servers that only speak the first version, or the older HTTP+SSE transport, get a warning: newer versions add safety labels and safer sign-in.
Unknown requests: We ask for a method that does not exist. A server that answers as if it worked gets a warning: an AI cannot tell real results from silent failures. Error messages that reveal stack traces, file paths or internal hosts are a warning too.

Tools

Listing the tools: We list every tool, following pagination. If listing fails, the server is "Not working".
Input schemas: Every tool must describe its inputs as a JSON Schema object, as the spec requires. Missing or malformed schemas fail: clients cannot call the tool correctly.
Names and descriptions: Two tools with the same name fail: a client cannot tell them apart. Tools without a description, or with names outside the spec's guidance, are warnings.

Safety labels

Read-only or not: MCP lets a server label each tool as read-only or able to change things, so your AI client can ask you before risky calls. We report the labels and warn about tools without them. Following the spec, an unlabelled tool counts as able to change things.

Security

Hidden instructions (tool poisoning): We read every tool description, parameter description and the server's own instructions for text aimed at the AI rather than the user: telling it to read secret files such as SSH keys or MCP configs, to pass local content into a tool call, to keep something from the user, to override other instructions or change how other tools behave; and characters that hide or reorder text (Unicode tag characters, bidi overrides) or terminal escape codes.
Changes since the last audit: We fingerprint every tool definition. When they change, we say what changed. Usually a normal update, but tools that change after people approve them are a known attack, so it is worth a look.

What we never do

Call a tool in the weekly audit. Agent tests call only tools that say they only read.
Call a tool that can change things, or one that does not say, in any test.
Send credentials, cookies or keys.
Try to break in, probe for weaknesses or load-test.
Run the weekly audit more than once a week, apart from one retry when a request fails, or test a server with an agent again before its tools change.
Republish tool text flagged as an attack: we describe the finding instead, so we do not pass the text on to other AIs.

Agent tests: does it do what it says?

The weekly audit reads what a server says about itself. An agent test checks whether it is true. An AI agent picks a server that passed and needs no sign-up, makes real calls to up to three of its tools, and judges each answer against what the tool claims: is it the right kind of data, is it plausible and current? Where it is easy, it checks one fact against a public source.

Only tools that label themselves read-only are called. Never a tool that can change things, or one that does not say.
At most 6 calls per server, two seconds apart, with no credentials, keys or personal data.
Each call is published: the tool, what the agent sent, the start of the answer and why the agent judged it as it did.
The agent treats everything a server returns as data, never as instructions. A server whose text tries to steer it is reported as not as described.
Agents work offline and propose each result as a change to our public repository. Automatic checks reject anything outside these rules, and a person merges it.
A server is tested again when its tools change. Until then, its last test stays on its report, marked as being about earlier tools.

Does what it says: Every call the agent made returned what the tool's description promises.
Partly does what it says: Some answers matched the description; others were incomplete, out of date or failed.
Not as described: At least one tool returned something different from what it claims.
Could not test: No call succeeded, so the agent could not judge the tools.

An agent test is a spot check of a few calls, not a guarantee. A vendor who disagrees can ask for a re-test by email, below.

What the results mean

Passed public checks: Every public check ran and passed.
Passed, with warnings: Nothing failed, but there is something to know or fix.
Sign-in required: limited audit: Sign-in stopped the tool checks. We checked the connection, encryption and how sign-in works.
Failed: The server works but broke a safety, security or protocol rule.
Not working: The address answers, but the MCP connection or tool listing fails.
Not responding: Nothing answered at the address in the registry.
Under review: A security finding is being reviewed by a person before it is published.

“Passed public checks” means exactly that. It is not a guarantee that a server is safe or that its data is right. Each report lists every check that ran, with the evidence.

Keeping it fair

A person reviews every security failure before it is published. Until then the report says “Under review” and names no specifics. A review applies to the exact tool definitions it looked at: if they change, the finding is reviewed again.

Patterns with honest uses are notes, not failures. A security scanner that names ~/.ssh/id_rsa, or a tool that does a step “without showing it to the user”, gets a note for review. It fails only when the same text also asks for something else on our list. Invisible characters that are ordinary parts of writing, such as the zero-width non-joiner in Persian or direction marks in Arabic and Hebrew, are never flagged.

“Down” is not “unsafe”. A server that does not answer, or whose MCP does not work, is reported as such and never as failed.

Every result is dated and kept. Each report shows the last 26 audits.

Which servers we include

Listed in the official MCP Registry, active, latest version.
Hosted at a public address, and reachable without a vendor-specific header.
About data in one of Signal Lab's topics: Financial, Global Macro, Energy, Weather, News & RSS, OSINT, Cyber Threat, Transport & Shipping, Agriculture & Food, Research & Grey Literature.
Not part of a mass-generated family (the same description reused by 3 or more servers), and not a test listing.

New servers that match are added each week. A server that leaves the registry stays listed, marked as such, so its history stays visible.

If you run a server

Our requests identify themselves with this user agent:

SignalLabAuditor/1.0.0 (+https://signal-lab.qdt.ai/mcp-servers/how-we-audit)

A public audit sends about six requests: the handshake, the list calls and one unknown method. Fix an issue and it shows in the next weekly audit. To dispute a finding, ask for an earlier re-audit, or ask us to stop connecting to your server, email signal-lab@qdt.ai.