Bring documented MCP tool names in line with the live servers - #90
Open
mattpodwysocki wants to merge 2 commits into
Open
mattpodwysocki wants to merge 2 commits into
mattpodwysocki wants to merge 2 commits into
Conversation
point_in_polygon_tool does not exist. The runtime MCP server calls it
points_within_polygon_tool — verified against a tools/list call. The old
name appeared 37 times across three skills, so any agent following these
docs would call a tool that is not there.
mapbox-mcp-runtime-patterns/AGENTS.md also disagreed with its own
references on two framework APIs:
- The LangChain example used DynamicTool; references/langchain.md uses
DynamicStructuredTool with a Zod schema, which is what keeps the model
from passing malformed coordinates.
- The Pydantic AI example passed a bare model string and gave
directions_tool `origin`/`destination` parameters. directions_tool takes
`coordinates`, a list of {longitude, latitude} objects, and the correct
import is OpenAIChatModel — OpenAIModel does not exist and raises
ImportError. The skill's own eval expectations already forbade the
origin/destination form that its AGENTS.md demonstrated.
Two isochrone_tool calls also passed `coordinates` positionally, hiding
that it takes a single {longitude, latitude} object.
Agents surface at n=5 moved 82% to 99% on this skill. Skill surface after
the rename: mcp-runtime 98.3%, geospatial-operations 95.8%.
Three further names are documented but absent from all three Mapbox MCP
servers: version_tool, create_token_tool and get_latest_mapbox_docs_tool.
Left alone here — unlike point_in_polygon_tool they have no confirmed
replacement, and guessing one would be worse than flagging them.
2 tasks done
Audited every *_tool name in the repo against tools/list on all three servers declared in .mcp.json — runtime (31 tools), devkit (22), docs (3). Four documented names existed on none of them, and two more were documented in a way that contradicts what the server now says. Runtime (mapbox-mcp-runtime-patterns): - version_tool is gone from the server. Removed from the tool table, the utility-ops list and examples/README.md. - category_list_tool is deprecated; its own description says to use resource_reader_tool with the mapbox://categories URI. Noted, and the tool-selection table now points at the replacement. - render_map_tool is undocumented, yet its description opens "PREFERRED MAP TOOL: ... ALWAYS use this tool to display maps". Added, with the distinction from static_map_image_tool: render_map_tool draws an interactive GL JS map and consumes the mapboxRender.ref other tools return, static_map_image_tool is for a static PNG or JPEG. - ground_location_tool and place_details_tool were also undocumented, as were nine local tools: convex, destination, difference, intersect, length, nearest_point, nearest_point_on_line, points_within_polygon and union. The table now lists all 17 local tools rather than 9. DevKit (mapbox-mcp-devkit-patterns): - create_token_tool does not exist. The skill told readers to run it as step 1 of token management, and an eval expectation required recommending it. Token creation is not an MCP operation — it happens in the account dashboard or through the Tokens API. list_tokens_tool, which does exist, inspects what is already there. - get_latest_mapbox_docs_tool does not exist on DevKit. Documentation lookup is a separate server, mcp-docs.mapbox.com, offering search_mapbox_docs_tool, get_document_tool and batch_get_documents_tool. Corrected and the endpoint named, since the skill pointed every MCP example at the DevKit URL. Scores after: devkit 96.3% skill / 98.1% agents, runtime 93.9% skill. The runtime figure is below an earlier 98.3% reading on the same content — that eval's runs were 100/67/100, so the difference is its own spread, not this change. Only call_tool and my_tool remain unmatched, and both are method names in example code rather than tool references.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #80, then extended: once the first wrong tool name turned up I audited every
*_toolin the repo againsttools/liston all three servers in.mcp.json— runtime (31 tools), devkit (22), docs (3).Four documented names exist on none of them, and two more are documented in a way the server now contradicts. An agent following these docs calls tools that aren't there.
Runtime —
mapbox-mcp-runtime-patternspoint_in_polygon_toolpoints_within_polygon_tool(37 occurrences, 3 skills)version_toolexamples/README.mdcategory_list_toolresource_reader_toolwithmapbox://categoriesrender_map_toolrender_map_tool's description opens "PREFERRED MAP TOOL: … ALWAYS use this tool to display maps", and the skill never mentioned it. Added with the distinction fromstatic_map_image_tool:render_map_tooldraws an interactive GL JS map and consumes themapboxRender.refother tools return instructuredContent;static_map_image_toolis for a static PNG or JPEG.Also undocumented and now listed:
ground_location_tool,place_details_tool, and nine local tools (convex,destination,difference,intersect,length,nearest_point,nearest_point_on_line,points_within_polygon,union). The table claimed 9 local tools; there are 17.DevKit —
mapbox-mcp-devkit-patternscreate_token_tooldoes not exist. The skill told readers to run it as step 1 of token management, and an eval expectation required recommending it. Token creation isn't an MCP operation — it happens in the account dashboard or via the Tokens API.list_tokens_tool, which does exist, inspects what's already there.get_latest_mapbox_docs_tooldoes not exist on DevKit. Documentation lookup is a separate server,mcp-docs.mapbox.com, offeringsearch_mapbox_docs_tool,get_document_toolandbatch_get_documents_tool. Corrected, and the endpoint named — the skill pointed every MCP example at the DevKit URL.AGENTS.mdalso disagreed with its own referencesDynamicTool;references/langchain.mdusesDynamicStructuredToolwith a Zod schema, which is what stops the model passing malformed coordinates.directions_toolorigin/destinationparameters. It takescoordinates, a list of{longitude, latitude}objects, and the import isOpenAIChatModel—OpenAIModeldoes not exist and raisesImportError. The skill's own eval expectations already forbade the form itsAGENTS.mddemonstrated.isochrone_toolcalls passedcoordinatespositionally, hiding that it takes a single{longitude, latitude}object.Measured
--repeats=3: devkit 96.3% skill / 98.1% agents; runtime 93.9% skill. Agents surface for runtime moved 82% → 99% in earlier measurement.The runtime figure sits below an earlier 98.3% reading on the same content. That eval's runs were 100/67/100, so the gap is its own spread rather than this change — which is the kind of thing the
--repeatswork in #87 exists to tell apart.Remaining
Only
call_toolandmy_toolare unmatched, and both are method names in example code rather than tool references.Ordering
Touches eval files that #87 relocates, so #87 should land first.
Test plan
npm run checkpasses*_toolname re-audited against livetools/liston all three servers — nothing unmatched but the two method namesmcp-devkit-patternsandmcp-runtime-patternsre-run at--repeats=3🤖 Generated with Claude Code