# Methodology — how the Discry Score works

> The Discry Score is a behavioral measurement of API agent-readiness: real models are quizzed on your live API docs and graded mechanically against citation-verified ground truth. Contamination-guarded, fame-decorrelated, plain-fetch, fully published.

Canonical: https://discry.ai/methodology · Markdown mirror: https://discry.ai/methodology.md

# How the Discry Score works.

AI agents are becoming a primary consumer of APIs. When a developer asks an agent to build something, the agent reads public documentation, decides which API it can operate, and quietly picks one. No sales call, no eval spreadsheet — just your docs, read by a model, in seconds. The Discry Score measures whether your API survives that moment.

There are two ways to lose it, and they happen in order. First: the agent never finds a usable version of your docs at all — no machine-readable entry point, or pages that render empty for anything that isn’t a browser. A remarkable share of APIs lose here, before their documentation quality is ever tested. Second: the agent finds your docs but can’t operate from them — it misreads your auth, guesses at your endpoints, or invents capabilities you don’t have. The score measures both, in that order: discovery, then comprehension.

## The score at a glance

- Discovery is graded first: can an agent find a usable, machine-legible version of your API at all — llms.txt, AGENTS.md, OpenAPI, MCP presence — on a plain fetch, the way agents actually arrive?
- Comprehension is graded by experiment: real models are quizzed on your live docs — factual questions, request construction, and trap tasks that probe for invented capabilities.
- Answers are graded mechanically against citation-verified ground truth, never by human judgment.
- A contamination guard removes anything a model already knew without your docs, so fame earns nothing.
- Every profile publishes its receipts: coverage, invalid runs, the URLs we fetched, and the tasks that failed.

## Two dimensions, one grade

Every published grade decomposes into the same two dimensions, so you can see where an API wins or loses:

- **DISCOVERY — Can an agent find you?** The machine-legible surface of your API: llms.txt, AGENTS.md, an OpenAPI spec, MCP manifests and registry presence, crawlable sitemaps — all fetched over plain HTTP, the way agents actually fetch. This is the gate: fail it and nothing else about your docs matters.
- **COMPREHENSION — Can an agent operate you?** The experiment: models across capability tiers answer quizzes with your fetched docs as their only source — find the facts, build correct requests, refuse what you don’t support. Graded mechanically against ground truth cited to your own pages.

## Discovery: the failure that happens first

Agents fetch over plain HTTP without executing JavaScript, and they navigate by machine-readable entry points. Discovery measures whether that path exists for your API: is there an llms.txt that actually routes to your reference, an AGENTS.md that briefs a coding agent, a spec it can parse, an MCP surface it can wire up? Serving these well is rewarded directly.

This is where a striking share of APIs — including famous ones — lose outright. Some documentation renders essentially nothing without JavaScript. Some llms.txt files point agents at marketing pages instead of the API reference. When that happens, the profile says so plainly, with the reason and the URLs we tried, because a discovery failure is a verifiable property of your public surface — and usually the cheapest thing on the page to fix.

### Isn’t MCP the answer?

Partly — and the score credits it. MCP manifests and registry presence earn discovery credit today, because a working MCP surface is a genuine machine-legible entry point.

But MCP narrows the problem; it doesn’t dissolve it. An agent still has to find and choose your server among alternatives — that is discovery again, one level up. The long tail of APIs has no MCP server and won’t soon. And an MCP server is built on the same documentation we grade: a server wrapping confusing docs inherits the confusion. Every Discry artifact is stamped with the surface it graded, and measuring MCP surfaces directly is on the instrument roadmap.

## Comprehension: the test you get to take if you’re found

Once an agent has your docs, the question becomes operational: can it actually work your API? We run that as an experiment rather than a checklist. Models are quizzed from published task banks — facts an integration depends on (auth, errors, pagination, limits), requests to construct, and trap tasks asking for things your API doesn’t support, where the right answer is a refusal rather than a confident invention. Every answer is graded mechanically against ground truth that must cite a specific location in your own documentation.

## See the results

The methodology is one page; the data is the product. Explore it wherever you work:

- [The Discry Index](https://discry.ai/index) — every scored API, ranked, with per-profile receipts
- [Scan your API free](https://discry.ai/scan) — probe your own discovery signals in seconds
- [Machine-readable](https://discry.ai/api/index.json) — the full index as JSON (CSV at /api/index.csv, this page at /methodology.md)

# The instrument in depth

## How a score is produced

1. **Plain fetch** — We fetch your docs surface the way most agents do: plain HTTP, no JavaScript execution. What renders is what gets graded.
2. **Published task banks** — Every API is quizzed from versioned, published task banks: factual questions about auth, errors, pagination and limits; request-construction tasks; and trap tasks probing whether a model invents capabilities your API doesn’t have.
3. **Models answer from your docs alone** — Models across capability tiers answer with your fetched docs as their only source. A separate closed-book pass establishes what they already knew without the docs.
4. **Mechanical grading** — Answers are graded mechanically against ground truth that is citation-verified — every ground-truth fact must trace to a specific location in your own documentation.
5. **Decomposition and receipts** — The result publishes as discovery + comprehension, with the profile’s coverage, invalid-run count, fetched URLs, and failed tasks alongside it.

## The plain-fetch policy

The corpus is fetched over plain HTTP with no JavaScript execution, because that is how most agents fetch. If your documentation only renders with JavaScript, an agent on a plain fetch sees almost nothing — and we grade what it sees. Serving llms.txt or markdown mirrors is rewarded: they are exactly the surfaces that make an API legible to agents.

When a docs surface yields essentially nothing readable on plain fetch, the profile publishes as docs-unreachable — a rule-based failing grade that ships with the specific reason and the URLs we tried, so the result is verifiable and fixable. And when an API publishes an llms.txt, that file is treated as its declared agent-facing entry point: if it routes agents to marketing pages instead of the reference, the grade reflects the surface the API itself chose to present.

## The contamination guard

Frontier models have read the famous APIs’ docs in training. Without a control, a famous API would score well even if its live docs were empty — the model answers from memory, not from the documentation. So every task first runs closed-book: anything a model can answer without your docs is excluded from the eligible set, and only genuinely doc-dependent tasks count.

In the extreme case, models know these docs by heart — nothing left to measure. Those APIs publish as insufficient-coverage rather than being handed an unearned grade.

## Fame is decorrelated — provably

The point of the guard is that the score measures documentation, not brand. We tested it directly: after contamination exclusion, the correlation between an API’s closed-book fame (how much models already know about it) and its Discry Score is approximately zero (Spearman ≈ 0, weighted-lift study). Unadjusted, that correlation is meaningfully positive. A famous API earns nothing from being famous, and an obscure API loses nothing for being obscure.

## Run validity and coverage — disclosed, not hidden

Errored or aborted runs never fold into a published score; profiles disclose their invalid-run count. Each profile also reports request-task coverage — how many of its five request-construction tasks produced gradeable capability extractions. When coverage is too thin to support a conclusion, the profile says indeterminate instead of guessing.

Alongside the grade, each profile reports a capability floor: the weakest model tier that can still operate the API from its docs — from haiku-ready down to frontier-only, not-agent-ready, or indeterminate. Every artifact is stamped with the instrument version and execution environment it ran under, verified by an environment-equivalence probe.

## Before publication, we audited ourselves

Before any grade publishes, the instrument itself gets audited at the trace level. The audit’s conclusion, quoted in full:

> Before publication we audited a stratified 8-API sample of the corrected instrument (2.4.x) at the trace level — every failed task in each API’s reference run was independently ruled as a genuine documentation failure or an instrument error, with the API’s live agent-facing surface (llms.txt) fetched as a receipt where it drove the result. No sampled grade was overturned. One failed cell in 25 (4%) traced to an instrument error (a ground-truth extraction miss on one API’s auth-scheme fact), down from 84% before the correction round. Every low grade in the sample decodes to a verifiable property of the API’s own documentation surface. Separately, mechanical checks across all 196 artifacts confirmed zero errored runs folded into published medians and zero remaining cases of a correct request graded as wrong.

> — Widened post-fix sample audit, 2026-07-18

## Known limitations — and the fix path for each

- **GraphQL paradigm fit** — The request-construction task banks are REST-shaped; GraphQL APIs fit the paradigm imperfectly today. Paradigm-aware task banks are on the instrument roadmap.
- **Residual request sparsity** — Some docs yield few gradeable request-construction tasks, leaving the capability floor thinly supported. This is disclosed per profile via the request-coverage line, and improved request targeting is on the instrument roadmap.
- **Docs-surface-only scope** — The score currently grades your documentation surface. MCP presence earns discovery credit (see “Isn’t MCP the answer?” above), but MCP and CLI surfaces are not yet behaviorally measured; every artifact is stamped with the surface it graded.

## Think your grade is wrong?

The appeals path is a rescan, not an argument: request an on-demand rescan of your API and the published profile updates from the fresh artifact. On-demand rescans land with the Index launch; until then, start from the free scan. [Scan your API — free](https://discry.ai/scan)

## An independent measurement, published in full

Discry helps teams get agent-ready — that is the business, and we’re not neutral about wanting every API on the Index to improve. So the measurement is built to be independent of us: grading is mechanical, the task banks are versioned and published, every profile carries its receipts, and the same instrument grades every API — including this site, which ships its own llms.txt, AGENTS.md, and markdown mirrors to the standard it grades others on. The only thing that moves a grade is the documentation itself, verified by rescan.

We publish everything — methodology, task banks, receipts, and the data itself in machine-readable form — because agent-readiness is an industry-wide problem, and an open benchmark moves it faster than a private one.