Skip to content

fix(llm): force reasoning_effort none for gpt-5.6-terra tool calls and tolerate role-less choices - #1859

Merged
mjnovice merged 2 commits into
mainfrom
fix/terra-tool-call-reasoning-effort
Aug 12, 2026
Merged

fix(llm): force reasoning_effort none for gpt-5.6-terra tool calls and tolerate role-less choices#1859
mjnovice merged 2 commits into
mainfrom
fix/terra-tool-call-reasoning-effort

Conversation

@mjnovice

Copy link
Copy Markdown
Contributor

Problem

Since the AvailableLlmModels default flipped to gpt-5.6-terra (SRE-636507 in alpha, SRE-639489 in prd), every LLM-as-judge evaluation using terra fails with:

Exception thrown by evaluator: 2 validation errors for ChatCompletion
choices.0.message.role      Field required [type=missing, input_value={'tool_calls': [{'id': …, 'summary': []}]}]
choices.0.finish_reason     Field required [type=missing, input_value={'index': 0, 'message': {…}}]

Two terra behaviors combine to cause this:

  1. Terra rejects function tools with any reasoning effort other than "none" ("Function tools with reasoning_effort are not supported for gpt-5.6-terra … set reasoning_effort to 'none'").
  2. With reasoning left on, its forced tool-call choice comes back without message.role and without finish_reason, which the strict ChatCompletion pydantic model rejects — so the judge errors out and the eval run scores 0%.

The C# backend was already fixed for the same signature in UiPath/Agents#6020 (send reasoning_effort: none for terra tool calls) and UiPath/Agents#5995 (accept role-less tool-call choices). This is the Python SDK mirror for the evaluator path (llm_as_judge_evaluatorUiPathLlmChatService.chat_completions).

Changes

  • UiPathLlmChatService.chat_completions: when the request carries tools and the model is gpt-5.6-terra, send reasoning_effort: "none" in the normalized request body. Tool-less requests keep the default behavior (mirror of Agents#6020).
  • ChatMessage.role defaults to "assistant" and ChatCompletionChoice.finish_reason becomes Optional[str], so a role-less tool-call choice parses instead of being rejected (mirror of Agents#5995).

Tests

  • terra + tools → request body contains reasoning_effort: "none"
  • terra without tools → no reasoning_effort
  • other models + tools → no reasoning_effort
  • role-less / finish_reason-less tool-call response parses; role defaults to assistant, tool call args intact

pytest tests/services/test_uipath_llm_integration.py tests/services/test_llm_service.py: all pass except test_basic_chat_completions_mocked, which also fails on a clean origin/main checkout in my environment (an EndpointManager discovery call leaks into the mock's call count) — pre-existing, unrelated.

🤖 Generated with Claude Code

…d tolerate role-less choices

gpt-5.6-terra rejects function tools combined with any reasoning effort
other than "none", and with reasoning left on it returns a forced
tool-call choice without message.role or finish_reason. The strict
ChatCompletion pydantic model then failed validation ("2 validation
errors for ChatCompletion: choices.0.message.role Field required,
choices.0.finish_reason Field required"), so every LLM-as-judge
evaluation using terra errored out and scored 0%.

Python-side mirror of UiPath/Agents#6020 and UiPath/Agents#5995
(SRE-636507 / SRE-639489):

- UiPathLlmChatService.chat_completions sends reasoning_effort "none"
  for gpt-5.6-terra when the request carries tools; tool-less requests
  keep the default behavior.
- ChatMessage.role defaults to "assistant" and
  ChatCompletionChoice.finish_reason is optional, so a role-less
  tool-call choice parses instead of being rejected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 12, 2026 15:58
@github-actions github-actions Bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-integrations labels Aug 12, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the uipath-platform LLM Gateway integration to handle gpt-5.6-terra tool-call quirks that currently break LLM-as-judge evaluations: it forces reasoning_effort: "none" for tool-bearing terra requests and relaxes response parsing to accept tool-call choices missing message.role and finish_reason.

Changes:

  • Force reasoning_effort: "none" only for gpt-5.6-terra requests that include tools.
  • Make response models tolerant of terra’s role-less / finish_reason-less tool-call choices (ChatMessage.role default, finish_reason optional).
  • Add integration tests covering the request-body normalization and the relaxed response parsing.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
packages/uipath-platform/tests/services/test_uipath_llm_integration.py Adds test coverage for terra tool-call reasoning_effort normalization and role/finish_reason tolerant parsing.
packages/uipath-platform/src/uipath/platform/chat/llm_gateway.py Relaxes response schema validation to tolerate role-less messages and missing finish_reason.
packages/uipath-platform/src/uipath/platform/chat/_llm_gateway_service.py Implements conditional reasoning_effort: "none" injection for tool-bearing gpt-5.6-terra requests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →

@sonarqubecloud

Copy link
Copy Markdown

@mjnovice
mjnovice merged commit 6350883 into main Aug 12, 2026
177 of 178 checks passed
@mjnovice
mjnovice deleted the fix/terra-tool-call-reasoning-effort branch August 12, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-integrations test:uipath-langchain Triggers tests in the uipath-langchain-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants