DataNexus is Poor to agents.

What we found

• —An AI crawler looking for DataNexus's docs is blocked at the door: the marketing/docs site ships the default Cloudflare-managed robots.txt that Disallows GPTBot, ClaudeBot, Google-Extended, CCBot, Amazonbot and every other major AI user-agent, with Content-Signal ai-train=no — a striking contradiction for a product whose entire pitch is 'Public Data Intelligence for AI Agents.'
• —Discovery files agents actually look for are absent: no llms.txt, no llms-full.txt, no sitemap.xml, no AGENTS.md in the public repo, and /.well-known/mcp.json 404s on the live domain (the repo carries a .well-known/mcp-manifest.json that is never served at a discoverable path).
• —Where it wins on discovery is the MCP ecosystem: DataNexus is listed across every major registry — Glama (AAA, position 1), Smithery (98/100), plus ToolRank, MCP Index, Socket and Skills Playground — so an agent browsing registries finds it instantly even though crawlers can't reach the site.
• —Comprehension is excellent: tool names follow a rigid {category}_{verb}_{noun} snake_case scheme (security_fetch_cve_detail, nonprofit_fetch_nonprofit_by_ein), descriptions are task-oriented ('single-call SHIP/CAUTION/BLOCK verdict'), and examples use real identifiers (CVE-2021-44228, EIN 46-5734087, patent US10702600B2) that an agent can copy verbatim.
• —The primary doc is the GitHub README, delivered as clean 'AI-Ready Markdown' with tool tables (name / purpose / source / auth), a rate-limit tier table, a documented error-status enum and a 7-example quickstart — genuinely built for agent consumption, which is why comprehension scores an A while discovery scores an F.

What to change

Prioritized by impact on discoverability. You (or your docs platform) deploy these — Discry never touches your API.

1. 01Stop blocking AI crawlers on the docs site. Replace the default Cloudflare-managed robots.txt with rules that Allow GPTBot, ClaudeBot, Google-Extended and PerplexityBot on / and /docs — an MCP product marketed to AI agents should be the last site to disallow them. This is the single highest-impact discovery fix.
2. 02Add an llms.txt (and ideally llms-full.txt) at the domain root: a concise, API-focused index linking to /docs, the 7 tool categories, rate limits and error handling. Given the docs are already well-structured, this is a quick win that would lift multiple discovery checks at once.
3. 03Serve the MCP manifest at the conventional path: expose /.well-known/mcp.json (or ai-plugin.json) on datanexusmcp.com — the repo already has a .well-known/mcp-manifest.json, so it just needs to be published where agents probe for it.
4. 04Add an AGENTS.md to the datanexusmcp/mcp-server repo so coding agents get first-class setup/usage context, and publish a sitemap.xml covering the homepage, /docs and /use-cases so the (once unblocked) docs pages are discoverable.
5. 05Deepen workflow and error-recovery docs: turn the quickstart prompts into 2-3 dedicated end-to-end guides that show tool chaining WITH failure handling (e.g. what to do on a DEGRADED or INVALID_PARAMS response, and how to react to the usage/upgrade_hint on rate-limit approach).