Read Tool & Reasoning Tokens from GenAI API instead of Catalog Metadata

[Sayan Shaw (sayanshaw24)]

Migrate tool/reasoning metadata from catalog to GenAI + optimize decode loop

Summary

Two goals:

1. Catalog v2 migration: Tool-call and reasoning token metadata (toolCallStart, toolCallEnd, reasoningStart, reasoningEnd) is being removed from the catalog. This PR sources it from GenAI's genai_config.json instead, via the new OgaTokenizer::GetBotTokenId() / GetEotTokenId() / GetBorTokenId() / GetEorTokenId() APIs, and writes the decoded strings into ModelInfo so all existing downstream consumers continue working.

2. Decode loop optimization: The current per-token detection of special tokens (double-decode + string.find()) is replaced with a single integer comparison against cached token IDs - the same pattern used for EOS detection.

Naming Convention

This PR adopts the new bot/eot/bor/eor naming convention introduced in this GenAI PR:

Abbreviation	Expansion	Purpose
bot	beginning of tool (call)	marks start of tool-call content
eot	end of tool (call)	marks end of tool-call content
bor	beginning of reasoning	marks start of reasoning/thinking content
eor	end of reasoning	marks end of reasoning/thinking content

Mirrors bos/eos/pad in both naming and access pattern (same Tokenizer class).

Performance

Before (per token): 2 tokenizer stream decodes + 2 string.find() + string comparison
After (per token): 1 integer comparison + 1 normal-stream decode

The updated path eliminates the special tokenizer stream from decode and keeps only the normal stream.
There is no runtime double-decode fallback in Decode().

Storage (follows EOS pattern)

Data	Storage	Consumer
eos_token_ids_	GenAIModelInstance	Decode() - filters EOS tokens
tag_info_ (IDs + strings)	GenAIModelInstance	Decode() - fast-path detection
String copies	ModelInfo.string_properties	BuildToolCallContext() -> ToolCallStreamAccumulator

Changes

GenAIModelInstance (genai_model_instance.h / .cc)

• New TagInfo struct with bot_id/eot_id/bor_id/eor_id + decoded strings
• GetTagInfo() - lazy-cached via std::call_once (same pattern as GetEosTokenIds())
  • Calls tokenizer_->GetBotTokenId() etc. (4 getter calls, one-time)
  • Decodes each valid ID through tokenizer_with_special_->Decode(&id, 1) to get the string (4 decodes, one-time)

OnnxChatGenerator::Decode() (onnx_chat_generator.cc)

• Always uses single-stream decode
• Compares token IDs against cached bot/eot/bor/eor IDs
• Returns cached marker strings for tag tokens
• Decodes normal tokens once via the normal tokenizer stream

Model::Load() (model.cc)

• After loading the GenAI model, enriches ModelInfo string properties from GetTagInfo() cached strings
• Only writes if the property isn't already set (catalog metadata takes precedence)
• Does not infer supports_tool_calling / supports_reasoning from tag ID presence

No changes to ToolCallContext, ToolCallStreamAccumulator, or BuildToolCallContext()

• They continue consuming strings from ModelInfo as before - single source of truth

Dependency

Requires updated onnxruntime-genai nuget with OgaTokenizerGetBotTokenId / GetEotTokenId / GetBorTokenId / GetEorTokenId APIs.

• GenAI PR: Add tool-calling and reasoning token IDs to GenAI Tokenizer (bot/eot/bor/eor)
• Foundry Local uses the newer ORT GenAI nightly containing these APIs.

How it works

Before (per token, every single generated token):

Decode Loop:
  token_id = generator.GetNextTokens()[0]

  // Decode through BOTH streams (expensive)
  token_text    = stream_->Decode(token_id)              // normal stream decode
  special_text  = stream_with_special_->Decode(token_id) // special stream decode

  // String operations to detect markers
  if (special_text != token_text):
    is_tool  = special_text.find("tool_call")            // string search
    is_think = special_text.find("think")                // string search
    if (is_tool || is_think): return special_text
  return token_text

Cost per token: 2 tokenizer stream decodes + 2 string.find() + string comparison - on every token, even though markers appear maybe 2-4 times per generation.

After (per token):

Model Load (one-time, 4 tokenizer getter calls + 4 Decode calls):
  GenAI:  tokenizer->GetBotTokenId() -> 151657   (from config or fallback vocab lookup)
  FL:     Decode(151657) -> "<tool_call>"           (one tokenizer decode, cached in TagInfo)
  FL:     ModelInfo["tool_call_start"] = "<tool_call>"

Decode Loop:
  token_id = generator.GetNextTokens()[0]

  // Integer comparison (virtually free)
  if (token_id == tag_info.bot_id) { stream_->Decode(token_id); return tag_info.bot_str; }
  if (token_id == tag_info.eot_id) { stream_->Decode(token_id); return tag_info.eot_str; }
  // ... bor/eor ...

  // Normal token - single decode, no special stream needed
  return stream_->Decode(token_id)

Cost per token: 1 integer comparison + 1 normal decode. No special stream. No string search.

Estimated performance gain

For a typical 500-token generation:

• Before: 500 x (2 decodes + 2 string searches) = 1000 decode calls + 1000 string ops
• After: 500 x (1 decode + 1 int compare) = 500 decode calls + 500 trivial comparisons

~50% reduction in tokenizer decode calls and complete elimination of string search operations in the hot loop. The special tokenizer stream is never even created/maintained for models with tag IDs configured.

Tag names supported by the GenAI API

• "tool_call_start", "tool_call_end" - tool call delimiters
• "reasoning_start", "reasoning_end" - chain-of-thought reasoning delimiters

Testing

Validated with local build and focused integration tests on the updated branch:

• C++ build succeeded (foundry_local_tests target)
• Tool-calling integration filter passed (9 tests)
• Response-format web service integration test passed (WebServiceIntegrationTest.ChatCompletionsWithResponseFormat)

The hot decode path now runs single-stream decode for all tokens.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
foundry-local	Ready	Preview	Sep 4, 2026 10:06pm UTC

[Copilot]

🟡 Changes recommended

The metadata data race and unresolved decode-path and behavior inconsistencies must be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Migrates tool/reasoning marker metadata from the catalog to GenAI tokenizer APIs and adds token-ID-based marker detection.

Changes:

• Caches GenAI tag IDs and decoded strings.
• Enriches model metadata during loading.
• Adds ID-based marker detection with legacy fallback.

File summaries

File	Review
sdk_v2/cpp/src/model.h	Makes stored model metadata mutable. No issue identified.
sdk_v2/cpp/src/model.cc	Critical (2 votes): Metadata mutation can race with unsynchronized readers. Moderate (2 votes): Support-flag behavior conflicts with the PR description. Nits (1 vote each): Add migration coverage; correct the qwen2 reasoning-marker example.
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.h	Defines cached tag metadata. No issue identified.
sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc	Nit (1 vote): Add coverage for caching, partial/unsupported IDs, enrichment, and decode paths.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.cc	Moderate (2 votes): The implementation still decodes through both streams, so the claimed optimization is not realized.

Review details

Suppressed comments (3)

sdk_v2/cpp/src/inferencing/generative/genai_model_instance.cc:104

• No test covers this new tag cache or its consumers. Existing C++ chat/model tests provide coverage for these areas, but none verifies configured versus unsupported/partial tag IDs, preservation of catalog values during enrichment, or marker emission through the fast and fallback decode paths. Add focused coverage before merging this metadata migration.

const GenAIModelInstance::TagInfo& GenAIModelInstance::GetTagInfo() {
  std::call_once(tag_info_init_flag_, [this]() {

sdk_v2/cpp/src/model.cc:524

• The central Catalog v2 behavior has no test asserting that a load populates missing tool/reasoning marker properties while preserving catalog-provided values. Existing model tests exercise Load() and metadata access, so add coverage for both cases to prevent this migration from silently regressing.

    const auto& tag_info = result.model->GetTagInfo();

sdk_v2/cpp/src/model.cc:540

• The fallback-map example is inaccurate: qwen2 has bot/eot fallback IDs, while bor/eor fallbacks are defined for qwen3. Please avoid citing qwen2 as a family with reasoning markers.

    // from the mere presence of tag token IDs.  The GenAI fallback map defines bor/eor
    // tokens for entire model families (e.g. all qwen2), but that doesn't mean every
    // variant actually supports reasoning (qwen2.5-0.5b is not a thinking model).

• Files reviewed: 5/5 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Baiju Meswani (baijumeswani)]

Thanks for this - the overall approach is clean and I like that it mirrors the existing EOS token pattern with call_once caching, keeps the slow path for backward compatibility, and deliberately avoids guessing the tool/reasoning support flags from the family-wide fallback map. Nice.

I left a few inline comments. The main one worth resolving before merge is the thread-safety of making ModelInfo writable and updating it inside Load(). The rest are smaller: the performance description doesn't quite match what the code does, a couple of description/code mismatches, and a small formatting nit.

A few non-blocking notes:

• Dependency: this needs the newer onnxruntime-genai package that has GetBotTokenId / GetEotTokenId / GetBorTokenId / GetEorTokenId. deps_versions.json isn't bumped in this PR, so CI won't build until that lands. Worth calling out in the PR so it isn't merged early.
• Tests: none here (understandable, since it needs the new package). Once the package is available, a small test that a model with these tokens fills in the tag strings, and a model without them still works via the old path, would give good coverage.
• Description mismatch: the "Changes -> Model::Load()" section says it "Infers supports_tool_calling / supports_reasoning from tag ID presence". The code intentionally does NOT do this (and the code comment explains why - good call). Please update the description so it matches.

[Sayan Shaw (sayanshaw24)]

Thanks for this - the overall approach is clean and I like that it mirrors the existing EOS token pattern with call_once caching, keeps the slow path for backward compatibility, and deliberately avoids guessing the tool/reasoning support flags from the family-wide fallback map. Nice.

I left a few inline comments. The main one worth resolving before merge is the thread-safety of making ModelInfo writable and updating it inside Load(). The rest are smaller: the performance description doesn't quite match what the code does, a couple of description/code mismatches, and a small formatting nit.

A few non-blocking notes:

• Dependency: this needs the newer onnxruntime-genai package that has GetBotTokenId / GetEotTokenId / GetBorTokenId / GetEorTokenId. deps_versions.json isn't bumped in this PR, so CI won't build until that lands. Worth calling out in the PR so it isn't merged early.
• Tests: none here (understandable, since it needs the new package). Once the package is available, a small test that a model with these tokens fills in the tag strings, and a model without them still works via the old path, would give good coverage.
• Description mismatch: the "Changes -> Model::Load()" section says it "Infers supports_tool_calling / supports_reasoning from tag ID presence". The code intentionally does NOT do this (and the code comment explains why - good call). Please update the description so it matches.

Thanks for the detailed review. Addressed the comments in the latest update:

• Thread safety: Restored ModelInfo immutability and removed the post-load mutation of info_->string_properties. Dynamically discovered GenAI marker strings are now read from the loaded model’s cached TagInfo and merged into the request-scoped ToolCallContext by ChatSession::BuildToolCallContext(). This avoids publishing mutable metadata and removes the data-race risk.
• Decode path: The implementation now uses a single tokenizer stream for every generated token. Tag IDs are compared directly, and matching tokens return their cached marker strings after advancing the normal stream. The special-token stream and double-decode path were removed.
• Capability flags: Support flags remain catalog-authored; tag-token presence is not used to infer tool-calling or reasoning support because the GenAI fallback map can apply to model families more broadly than the actual capability.
• PR description: Updated to match the immutable metadata ownership and single-stream decode behavior.
• Validation: The Windows C++ build passed, the focused tool-calling suite passed 9/9, and WebServiceIntegrationTest.ChatCompletionsWithResponseFormat passed.

The changes are pushed in commit d55c5ee (Avoid mutating published model metadata).