Skip to content

Bring documented MCP tool names in line with the live servers - #90

Open
mattpodwysocki wants to merge 2 commits into
mainfrom
fix-mcp-tool-accuracy
Open

mattpodwysocki wants to merge 2 commits into
mainfrom
fix-mcp-tool-accuracy

Conversation

@mattpodwysocki

@mattpodwysocki mattpodwysocki commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Split out of #80, then extended: once the first wrong tool name turned up I audited every *_tool in the repo against tools/list on all three servers in .mcp.json — runtime (31 tools), devkit (22), docs (3).

Four documented names exist on none of them, and two more are documented in a way the server now contradicts. An agent following these docs calls tools that aren't there.

Runtime — mapbox-mcp-runtime-patterns

name status fix
point_in_polygon_tool gone → points_within_polygon_tool (37 occurrences, 3 skills)
version_tool gone removed from the table, utility-ops list, and examples/README.md
category_list_tool deprecated its own description says use resource_reader_tool with mapbox://categories
render_map_tool undocumented added — see below

render_map_tool's description opens "PREFERRED MAP TOOL: … ALWAYS use this tool to display maps", and the skill never mentioned it. Added with the distinction from static_map_image_tool: render_map_tool draws an interactive GL JS map and consumes the mapboxRender.ref other tools return in structuredContent; static_map_image_tool is for a static PNG or JPEG.

Also undocumented and now listed: ground_location_tool, place_details_tool, and nine local tools (convex, destination, difference, intersect, length, nearest_point, nearest_point_on_line, points_within_polygon, union). The table claimed 9 local tools; there are 17.

DevKit — mapbox-mcp-devkit-patterns

  • create_token_tool does not exist. The skill told readers to run it as step 1 of token management, and an eval expectation required recommending it. Token creation isn't an MCP operation — it happens in the account dashboard or via the Tokens API. list_tokens_tool, which does exist, inspects what's already there.
  • get_latest_mapbox_docs_tool does not exist on DevKit. Documentation lookup is a separate server, mcp-docs.mapbox.com, offering search_mapbox_docs_tool, get_document_tool and batch_get_documents_tool. Corrected, and the endpoint named — the skill pointed every MCP example at the DevKit URL.

AGENTS.md also disagreed with its own references

  • LangChain used DynamicTool; references/langchain.md uses DynamicStructuredTool with a Zod schema, which is what stops the model passing malformed coordinates.
  • Pydantic AI passed a bare model string and gave directions_tool origin/destination parameters. It takes coordinates, a list of {longitude, latitude} objects, and the import is OpenAIChatModel — OpenAIModel does not exist and raises ImportError. The skill's own eval expectations already forbade the form its AGENTS.md demonstrated.
  • Two isochrone_tool calls passed coordinates positionally, hiding that it takes a single {longitude, latitude} object.

Measured

--repeats=3: devkit 96.3% skill / 98.1% agents; runtime 93.9% skill. Agents surface for runtime moved 82% → 99% in earlier measurement.

The runtime figure sits below an earlier 98.3% reading on the same content. That eval's runs were 100/67/100, so the gap is its own spread rather than this change — which is the kind of thing the --repeats work in #87 exists to tell apart.

Remaining

Only call_tool and my_tool are unmatched, and both are method names in example code rather than tool references.

Ordering

Touches eval files that #87 relocates, so #87 should land first.

Test plan

  • npm run check passes
  • Every *_tool name re-audited against live tools/list on all three servers — nothing unmatched but the two method names
  • mcp-devkit-patterns and mcp-runtime-patterns re-run at --repeats=3

🤖 Generated with Claude Code

point_in_polygon_tool does not exist. The runtime MCP server calls it
points_within_polygon_tool — verified against a tools/list call. The old
name appeared 37 times across three skills, so any agent following these
docs would call a tool that is not there.

mapbox-mcp-runtime-patterns/AGENTS.md also disagreed with its own
references on two framework APIs:

- The LangChain example used DynamicTool; references/langchain.md uses
  DynamicStructuredTool with a Zod schema, which is what keeps the model
  from passing malformed coordinates.
- The Pydantic AI example passed a bare model string and gave
  directions_tool `origin`/`destination` parameters. directions_tool takes
  `coordinates`, a list of {longitude, latitude} objects, and the correct
  import is OpenAIChatModel — OpenAIModel does not exist and raises
  ImportError. The skill's own eval expectations already forbade the
  origin/destination form that its AGENTS.md demonstrated.

Two isochrone_tool calls also passed `coordinates` positionally, hiding
that it takes a single {longitude, latitude} object.

Agents surface at n=5 moved 82% to 99% on this skill. Skill surface after
the rename: mcp-runtime 98.3%, geospatial-operations 95.8%.

Three further names are documented but absent from all three Mapbox MCP
servers: version_tool, create_token_tool and get_latest_mapbox_docs_tool.
Left alone here — unlike point_in_polygon_tool they have no confirmed
replacement, and guessing one would be worse than flagging them.
Audited every *_tool name in the repo against tools/list on all three
servers declared in .mcp.json — runtime (31 tools), devkit (22), docs (3).
Four documented names existed on none of them, and two more were
documented in a way that contradicts what the server now says.

Runtime (mapbox-mcp-runtime-patterns):

- version_tool is gone from the server. Removed from the tool table, the
  utility-ops list and examples/README.md.
- category_list_tool is deprecated; its own description says to use
  resource_reader_tool with the mapbox://categories URI. Noted, and the
  tool-selection table now points at the replacement.
- render_map_tool is undocumented, yet its description opens "PREFERRED
  MAP TOOL: ... ALWAYS use this tool to display maps". Added, with the
  distinction from static_map_image_tool: render_map_tool draws an
  interactive GL JS map and consumes the mapboxRender.ref other tools
  return, static_map_image_tool is for a static PNG or JPEG.
- ground_location_tool and place_details_tool were also undocumented, as
  were nine local tools: convex, destination, difference, intersect,
  length, nearest_point, nearest_point_on_line, points_within_polygon and
  union. The table now lists all 17 local tools rather than 9.

DevKit (mapbox-mcp-devkit-patterns):

- create_token_tool does not exist. The skill told readers to run it as
  step 1 of token management, and an eval expectation required
  recommending it. Token creation is not an MCP operation — it happens in
  the account dashboard or through the Tokens API. list_tokens_tool, which
  does exist, inspects what is already there.
- get_latest_mapbox_docs_tool does not exist on DevKit. Documentation
  lookup is a separate server, mcp-docs.mapbox.com, offering
  search_mapbox_docs_tool, get_document_tool and batch_get_documents_tool.
  Corrected and the endpoint named, since the skill pointed every MCP
  example at the DevKit URL.

Scores after: devkit 96.3% skill / 98.1% agents, runtime 93.9% skill. The
runtime figure is below an earlier 98.3% reading on the same content —
that eval's runs were 100/67/100, so the difference is its own spread, not
this change.

Only call_tool and my_tool remain unmatched, and both are method names in
example code rather than tool references.
@mattpodwysocki mattpodwysocki changed the title Correct MCP tool names and framework APIs in mcp-runtime patterns Bring documented MCP tool names in line with the live servers Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant