does-it-install?

Methodology

How a green, amber or red square on this site is produced, and what it does not tell you.

What one probe does

Every server in the catalog is probed independently, in a disposable CI runner, with no shared state between servers. The sweep picks the first distribution it supports (npm, then PyPI, then a container image, then a hosted endpoint) and walks these phases:

A server only shows green when every phase succeeded. Otherwise the status names the phase that broke, or says the server asked for something we do not have, and the newest stderr output is kept and shown verbatim on the server page either way.

A PyPI package rarely names the console script it installs, so the probe guesses it from the package name. When that guess is wrong, uv normally names the executable it did find, and the probe re-runs once with that name: an entry point named differently from its package now passes instead of showing a red handshake failure. The guess only survives as a limitation when uv gives no hint at all.

Time budgets

Each phase has its own budget: 600s to install, 30s to spawn, 30s for the handshake, 15s for tools/list, and 20s to reach a hosted endpoint. Exceeding one records timed out against that phase rather than a generic failure.

Probes run four at a time, so an install that ran out of time is probed once more on its own, with double the install budget, before the result stands. Timed out here therefore means the server exceeded its budget with the runner to itself. When a server passes but took more than a minute to install, its platform block says how long that took: a pass after ten minutes of installing is a different experience from a pass after twenty seconds.

Credentials and configuration

We probe with no accounts anywhere. Environment variables a server declares as required are filled with the literal placeholder DOES_IT_INSTALL_PLACEHOLDER, and every affected server page says so above its results. A server that needs a real API key can therefore install perfectly and still refuse to start or to finish the handshake here. That outcome is recorded as needs credentials, in amber, rather than as a failure: the server declared what it wanted, and we did not bring it.

Arguments work the same way. A registry entry sometimes declares arguments with no concrete value, a workspace path or an account id only the user knows, and the catalog drops those rather than invent them. A server that installs and then will not start or will not finish its handshake without them is recorded needs configuration, also in amber. When an entry declared both placeholder variables and placeholder arguments, the result reads needs credentials: a missing key is the commoner cause, and one phrase per platform is enough.

Going green (for maintainers)

The amber state resolves without anyone sharing a secret, and the fix is ordinary good server design: validate credentials lazily. Start, complete the initialize handshake, answer tools/list, and return a clear error from the tool call itself when the key is missing or wrong. A server built that way turns green here on its own, and real users get a visible tool list plus an error in context instead of a silent connection failure. Hosted servers can do the same over HTTP: allow anonymous initialize and tools/list, gate the tool calls. If your tools genuinely cannot be listed without a tenant, the standard OAuth challenge (a 401 with WWW-Authenticate) is correct, and amber is your accurate steady state: alive, gated, working as designed.

The registry entry does half the work. Declaring the environment variables and the arguments a server actually requires is what lets a probe tell "it asked us for something" from "it broke", so an accurate entry is the difference between an amber result and a red one. A server that silently requires a key or a flag it never declared gets no benefit of the doubt, because there is nothing to give it.

We do not accept test credentials, from anyone. The harness executes third-party code weekly, so holding real secrets would make every probe a liability, and probing with no accounts anywhere is what keeps the results comparable.

Platforms and cadence

Probes run on Linux, macOS, Windows runners, weekly, plus manual re-runs. Container images are probed on Linux only, because the macOS and Windows runners cannot run Linux containers; on those platforms the result is not tested, and a platform we did not test is left out of the overall badge instead of counting against the server. Each server keeps its last 30 results per platform, which is what the history strip shows. Servers that disappear from the registry keep their pages: knowing when something stopped working is the point.

Caveats

Corrections

Every result is reproducible from the command shown on the server page. If a result looks wrong, the error excerpt is the whole evidence we have, so file an issue on the project repository with it.