Add token-aware reasoning output

[Baiju Meswani (baijumeswani)]

Summary

Adds token-aware reasoning output to the C++ SDK, separating model reasoning from visible output and improving reasoning/tool-call streaming.

• Introduces a token-aware ReasoningStreamSplitter that recognizes reasoning boundaries from token IDs, with decoded <think>...</think> matching as a fallback.
• Handles empty-decoded special tokens, multi-token markers, truncated reasoning, newline trimming, and EOS/control tokens.
• Preserves ordered reasoning, visible text, and tool-call events without allowing tool-call-shaped reasoning to execute as a tool call.
• Tracks reasoning token counts and exposes them through Chat Completions (usage.completion_tokens_details.reasoning_tokens) and the Responses API (usage.output_tokens_details.reasoning_tokens).
• Represents reasoning as typed output in the Responses API and streams it separately from visible text.
• Adds complete Responses API function-call streaming events, including argument delta/done events.
• Preserves invalid tool-call blocks as visible text rather than silently dropping them.
• Preserves an assistant history entry for reasoning-only turns, preventing consecutive user roles on the next request.
• Adds unit coverage for reasoning boundaries, token accounting, interleaving, truncation, tool calls, and response conversion.

Remaining gap

/v1/chat/completions still intentionally drops reasoning segments. The current implementation does not define or emit delta.reasoning_content; only visible text is sent through delta.content. The PR therefore provides separate reasoning output through the Responses API, but does not yet provide the Copilot/DeepSeek-compatible Chat Completions streaming shape.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview	Aug 27, 2026 11:35pm

[David Fan (jiafatom)]

The internal token-aware split and Responses API reasoning events look good. For equivalent separation in OpenAI-compatible Chat Completions, could we add an optional reasoning_content member to ChatCompletionDelta, serialize it when present, and emit reasoning segments through that field instead of dropping them?

The expected reasoning chunk would be:

{
  "choices": [{
    "delta": {
      "reasoning_content": "Analyzing project files..."
    }
  }]
}

followed by visible output through:

{
  "choices": [{
    "delta": {
      "content": "I found three issues."
    }
  }]
}

Tests should cover reasoning-only and interleaved reasoning/content chunks so ordinary models continue emitting only delta.content.

[David Fan (jiafatom)]

Could we stream these segments through delta.reasoning_content instead of dropping them? The splitter already provides the necessary classification, and emitting a separate field would let Copilot/VS Code and other OpenAI-compatible clients hide, collapse, or style reasoning independently while preserving visible output in delta.content.

[Copilot]

Pull request overview

Adds token-aware reasoning segmentation, token accounting, ordered tool-call handling, and API serialization across the C++ SDK.

Changes:

• Detects reasoning boundaries using token IDs with text fallback.
• Preserves ordered reasoning, visible text, and tool-call events.
• Exposes reasoning-token usage and adds unit coverage.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sdk_v2/cpp/src/contracts/chat_completions_converter.cc	Filters reasoning and maps usage.
sdk_v2/cpp/src/inferencing/generative/chat/chat_generator.h	Exposes current token IDs.
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.cc	Integrates reasoning and ordered events.
sdk_v2/cpp/src/inferencing/generative/chat/chat_session.h	Defines generated event handling.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.cc	Captures generated token IDs.
sdk_v2/cpp/src/inferencing/generative/chat/onnx_chat_generator.h	Stores current token state.
sdk_v2/cpp/src/inferencing/generative/chat/reasoning_stream_splitter.h	Implements token-aware splitting.
sdk_v2/cpp/src/inferencing/generative/openresponses/response_converter.cc	Converts typed reasoning output.
sdk_v2/cpp/src/inferencing/generative/preprocessor.cc	Adds token-ID encoding.
sdk_v2/cpp/src/inferencing/generative/preprocessor.h	Declares token-ID encoding.
sdk_v2/cpp/src/inferencing/generative/toolcalling/tool_call_stream_accumulator.h	Adds ordered accumulator events.
sdk_v2/cpp/src/inferencing/session/response.h	Tracks reasoning-token usage.
sdk_v2/cpp/src/service/chat_completions_handler.cc	Streams reasoning usage.
sdk_v2/cpp/src/service/responses_handler.cc	Streams tool-call output items.
sdk_v2/cpp/test/CMakeLists.txt	Registers splitter tests.
sdk_v2/cpp/test/internal_api/chat/reasoning_stream_splitter_test.cc	Tests splitting and counting.
sdk_v2/cpp/test/internal_api/chat_completions_converter_test.cc	Tests filtered reasoning output.
sdk_v2/cpp/test/internal_api/chat_completions_test.cc	Tests usage serialization.
sdk_v2/cpp/test/internal_api/response_converter_test.cc	Tests ordered reasoning conversion.
sdk_v2/cpp/test/internal_api/response_store_test.cc	Tests stored reasoning usage.
sdk_v2/cpp/test/internal_api/toolcalling/tool_call_stream_accumulator_test.cc	Tests ordered tool events.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.