Token
A token is the unit of text a language model reads and generates — a word fragment averaging roughly four characters of English for common tokenizers — and the unit in which model usage is priced and context windows are sized. When an agent ingests API documentation, every character it reads is paid for in tokens, in both money and window space.
Models process subword vocabularies rather than words: common words map to single tokens, rare strings split into several, and markup, whitespace, and boilerplate all count. Input and output are both metered. The practical consequence for documentation is that format changes token cost dramatically — the same API reference as clean markdown can cost a fraction of what its JavaScript-rendered HTML equivalent costs to ingest.
At agent scale the cost compounds. A docs surface read once by a human is read thousands of times by agents scaffolding integrations, and verbose or markup-heavy documentation taxes every one of those reads. Token efficiency is therefore an economic property of docs, and it interacts with capability: the more of the window the docs consume, the less remains for the agent's actual task.
Token counts appear directly in Discry's measurement: the llms-full.txt discovery check reports the token size of the full-text bundle, and the comprehension dimension includes a token-efficiency check on whether the docs are concise enough to fit an agent context window.
What does an AI agent make of your API?
Find out in about a minute — no signup.
Discry your API — free