Apply guidance for user-specified response_format regardless of tool_choice

[Sayan Shaw (sayanshaw24)]

Fix: Apply guidance for user-specified response_format regardless of tool_choice

Fixes #1042

Problem

response_format with json_schema was silently ignored — the model generated freely without schema constraints. This affected all tool_choice modes except "required".

Root cause: SetGuidance() was gated behind tool_call_only, which is only true when tool_choice = "required":

bool tool_call_only = tool_ctx.tool_output && !tool_ctx.text_output;
if (!guidance_type.empty() && !guidance_data.empty() && tool_call_only) {
    gen_params->SetGuidance(...);  // never reached for auto/none
}

The guard was added to prevent a CoT multi-turn issue with auto-generated tool grammar, but it inadvertently blocked ALL guidance — including user-explicit response_format.

Fix

Separate user-explicit guidance from auto-generated tool grammar:

bool user_specified_guidance = !tool_ctx.guidance_type.empty() && !tool_ctx.guidance_data.empty();
bool tool_call_only = tool_ctx.tool_output && !tool_ctx.text_output;

if (!guidance_type.empty() && !guidance_data.empty() && (user_specified_guidance || tool_call_only)) {
    gen_params->SetGuidance(...);
}

• User-explicit guidance (response_format) → always applied
• Auto-generated tool grammar → only applied in tool-call-only mode (original guard preserved)

Also updated the cache invalidation check in chat_session.cc to account for user-specified guidance changes between turns.

Changes

File	Change
onnx_chat_generator.cc	Separate guidance gate: user-explicit always applies, auto-generated tool grammar gated by tool_call_only
chat_session.cc	Cache invalidation now considers user-specified guidance changes (not just tool_call_only)
chat_completions_test.cc	New test: ChatCompletionsWithJsonSchemaGuidance — verifies model output conforms to json_schema when response_format is set

Testing

• All existing tests pass (tool calling, chat, streaming)
• New test verifies json_schema guidance produces valid JSON with correct field types

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview	Aug 27, 2026 10:08pm

[Copilot]

Pull request overview

Applies explicit structured-output guidance independently of tool-choice mode.

Changes:

• Applies user-provided guidance unconditionally.
• Updates cached-generator guidance handling.
• Adds JSON-schema integration coverage.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
onnx_chat_generator.cc	Separates explicit guidance from tool grammar.
chat_session.cc	Updates guidance-related cache invalidation.
chat_completions_test.cc	Tests JSON-schema-constrained output.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.