Skip to content

[Inference] Fix ChatGPT 6 reasoning - #306548

Open
sphilipse wants to merge 9 commits into
elastic:mainfrom
sphilipse:fix-chatgpt-56-reasoning
Open

sphilipse wants to merge 9 commits into
elastic:mainfrom
sphilipse:fix-chatgpt-56-reasoning

Conversation

@sphilipse

@sphilipse sphilipse commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

OpenAI Chat Completions rejects function tools on newer reasoning models unless reasoning effort is explicitly none:

This makes those models unusable for tool calling (Agent Builder, Workflows) on any path that calls Chat Completions directly. The inference plugin already defaults to reasoning: { effort: 'none' } for gpt-5* models on inference endpoints. This PR closes the remaining gaps.

Behaviour

effort: 'none' is only sent when native function tools are present. "Listed models" are matched by name, because neither path has capability metadata to check against.

Path Listed models With native tools Caller reasoning
OpenAI connector (incl. Azure) gpt-5.6*, gpt-6-luna*, gpt-6-sol* Always reasoning_effort: 'none' (debug log if this overrides the caller) Not forwarded, as on main
Inference endpoint / .inference connector, non-EIS gpt-5*, gpt-6-luna*, gpt-6-sol* Default reasoning: { effort: 'none' } Forwarded, wins over the default, as on main
Inference endpoint / .inference connector, EIS n/a No default: EIS accepts tools at any effort Forwarded

Every other model gets no reasoning parameter unless the caller set one on an inference endpoint path.

For the OpenAI connector, the model is the request's modelName or the connector's defaultModel. An Azure deployment name is not used, so an Azure connector without a configured model gets no reasoning_effort.

Out of scope

  • Forwarding the caller's reasoning effort from the OpenAI connector. It needs to know which models accept reasoning_effort, which the connector has no source for.
  • Making none win over an explicit caller effort for the listed models on non-EIS inference endpoints. Same rule as the connector, but it changes existing behaviour.
  • Narrowing the existing gpt-5* entry on the inference endpoint path.
  • Replacing the model name lists, which needs the Responses API or parameter capabilities exposed by Elasticsearch.

sphilipse and others added 3 commits November 19, 2026 15:05
…th tools

OpenAI Chat Completions rejects function tools on gpt-5.6 and gpt-6 models
unless reasoning effort is explicitly `none`, which made those models
unusable for tool-calling features such as Agent Builder and Workflows.

- Add `gpt-6` to the models that default to `reasoning: { effort: 'none' }`
  when native tools are present on the inference endpoint and `.inference`
  connector paths. The existing `gpt-5` entry already covered gpt-5.6.
- Send `reasoning_effort: 'none'` from the native OpenAI connector adapter
  for gpt-5.6 and gpt-6 models when native tools are present. That adapter
  previously sent no reasoning effort at all. The list is narrower than the
  inference endpoint one because older gpt-5 models reject `none` when
  called directly.
- Widen `OpenAIRequest.reasoning_effort` with `none`, which the `openai`
  SDK types omit.
- Cover both paths with unit tests.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…t support it

Adding `gpt-6` to the models that default to `reasoning: { effort: 'none' }`
with native tools broke EIS endpoints for gpt-6 models that do not accept
that level (gpt-6-astra, gpt-6-astra-pro, gpt-6.1-sol). They return a 400
for `none` and work when reasoning is omitted.

- Skip the default when the endpoint advertises supported reasoning effort
  levels that do not include `none`. Explicit caller values still win, and
  the default still applies when no levels are advertised.
- Resolve the advertised levels from the inference endpoint metadata and
  pass them through the inference endpoint adapter.
- Apply the same check for the `.inference` connector using the connector
  metadata.
- Cover the resolver, endpoint resolution and endpoint adapter with tests.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…wn models

The native OpenAI connector sent `reasoning_effort: 'none'` with native
tools for any gpt-6 model. Some gpt-6 models (gpt-6-astra, gpt-6-astra-pro,
gpt-6.1-sol) reject `none`, and the connector has no capability metadata
to check against.

- Limit the default to gpt-5.6, gpt-6-luna and gpt-6-sol models, which
  accept `none`.
- Extend the resolver tests with the newly included and excluded models.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor
🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

  • Click to trigger kibana-pull-request for this PR!
  • Click to trigger kibana-deploy-project-from-pr for this PR!
  • Click to trigger kibana-deploy-cloud-from-pr for this PR!
  • Click to trigger kibana-entity-store-performance-from-pr for this PR!
  • Click to trigger kibana-storybooks-from-pr for this PR!
  • Click to trigger kibana-static-checks for this PR!

@sphilipse sphilipse removed the v9.5.0 label Nov 19, 2026
sphilipse and others added 2 commits November 19, 2026 17:07
…zure deployments

EIS returns tool calls for gpt-5 and gpt-6 models with reasoning omitted,
so the `effort: none` default is not needed there. Sending it turned
reasoning off for models that work at their default effort, and returned
a 400 for models that do not accept `none` (gpt-6-astra, gpt-6.1-sol).

- Skip the default when the inference endpoint or `.inference` connector
  provider is `elastic`. Explicit caller values are still forwarded.
- Remove the advertised effort level check and its plumbing. Those levels
  are only advertised by EIS, so the check is superseded.
- Resolve the model from the deployment name in the URL for Azure OpenAI
  connectors that have no model configured, so they get the
  `reasoning_effort: 'none'` default for the models that require it.
- Update the tests that pinned the default for EIS and cover the new
  behaviour.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ept it

The inference endpoint default matched the whole gpt-6 family. A non-EIS
endpoint for a model such as gpt-6-astra or gpt-6.1-sol would receive
`reasoning: { effort: 'none' }` with native tools, which those models
reject.

- Limit the gpt-6 entries to gpt-6-luna and gpt-6-sol, matching the list
  used for the native OpenAI connector.
- Add the rejecting gpt-6 models to the resolver tests.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
sphilipse and others added 2 commits November 20, 2026 10:58
An Azure deployment name is a user-defined alias, not a model ID. Using
it to decide whether to send `reasoning_effort: 'none'` had two problems:

- A deployment named after a model that requires `none` but backed by
  another model (e.g. `gpt-6-luna-prod` on `gpt-4o`) started receiving
  `reasoning_effort: 'none'`, which can break tool calls that worked
  before.
- A deployment of such a model with an unrelated name still got no
  default.

The OpenAI adapter now resolves the model only from the request's
`modelName` or the connector's `defaultModel`, which Azure connectors can
set. Azure connectors without a configured model get no default.

- Remove `getAzureDeploymentName` from the OpenAI adapter.
- Replace the deployment-name tests with one that covers an Azure
  connector with `defaultModel` and one that pins that the deployment
  name is not used.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kibanamachine

Copy link
Copy Markdown
Contributor

PR run: bk-01a1f913-6d65-4548-ac03-e7c7d21021a7::smoke-tests::anthropic-claude-4.5-haiku | Baseline (main): bk-01a1e8e2-ad19-43f7-a122-d7bfe80dda24::smoke-tests::anthropic-claude-4.5-haiku
Baseline: commit e2de501, 3 days ago

Warning: Baseline is 3 days old. Results may not reflect current main.
Significance threshold: p < 0.05
Repetition policy: mean (N counts examples).
Test per row is chosen from the scores: McNemar when every compared score is 0 or 1, otherwise Wilcoxon signed-rank (paired t-test for continuous scores when n ≥ 30 and differences look normal).

Summary
No significant regressions detected (6 evaluator comparisons).

View full comparison in UI | Refresh baseline against latest main (click Unblock in the eval build)

No significant changes (6 rows)
Dataset Evaluator N Mean (PR) Mean (main) Diff Test p-value Sig Outcome
smoke tests: es-snapshot-loader SnapshotRestored 1 1.00 1.00 0.00 (0 target only, 0 baseline only) McNemar 1.00 No -
smoke tests: llm-judge Criteria 1 1.00 1.00 0.00 (0 target only, 0 baseline only) McNemar 1.00 No -
smoke tests: score ingestion and code evaluator ContainsKibana 1 1.00 1.00 0.00 (0 target only, 0 baseline only) McNemar 1.00 No -
smoke tests: trace-retrieval Estimated Cost (USD) 1 0.00 0.00 0.00 Wilcoxon 1.00 No -
smoke tests: trace-retrieval Input Tokens 1 12.00 12.00 0.00 Wilcoxon 1.00 No -
smoke tests: trace-retrieval Output Tokens 1 4.00 4.00 0.00 Wilcoxon 1.00 No -

@infra-vault-gh-plugin-prod

Copy link
Copy Markdown
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
kbnUiSharedDeps-srcJs 5.5MB 5.5MB +4.0B
vendors-heavy 2.0MB 2.0MB +8.0B
total +12.0B
Unknown metric groups

total optimizer output size

id before after diff
all 69.2MB 69.2MB +8.0B

warm start memory

id before after diff
post forced gc heap baseline - 629072158 +629072158
post forced gc heap delta - -2736686 -2736686
post forced gc heap delta standard deviation - 3408595 +3408595
post forced gc heap target - 626335472 +626335472
tail heap delta - -17880200 -17880200
total +1238199339

workflow yaml validation

id before after diff
infosec_demo.yaml (150 steps, 270 vars)/e2e/performComputation 53 56 +3
infosec_demo.yaml (150 steps, 270 vars)/e2e/total 146 121 -25
infosec_demo.yaml (150 steps, 270 vars)/e2e/validateJsonSchemaDefaults 2 1 -1
infosec_demo.yaml (150 steps, 270 vars)/e2e/validateVariables 49 43 -6
infosec_demo.yaml (150 steps, 270 vars)/validateIfConditions 7 4 -3
infosec_demo.yaml (150 steps, 270 vars)/validateVariables (270 vars) 46 44 -2
total -34

Test Failures

  • [job] [logs] FTR Configs #74 / Alerting builtin alertTypes index_threshold rule runs correctly: count all < >
  • [job] [logs] Scout Lane #15 - stateful-classic / default / local-stateful-classic - Network flyout v2 - clicking the source.ip value in the alerts table opens the network flyout v2

History

…nector adapter

The OpenAI connector adapter never read the caller's `reasoning`, so an
explicit effort (e.g. a Workflow step set to `high`) was ignored. With
the `reasoning_effort: 'none'` default for gpt-5.6, gpt-6-luna and
gpt-6-sol, such a caller was silently overridden with `none`.

- Pass `reasoning` to `resolveOpenAIReasoningEffort`. An explicit effort
  always wins and is sent as `reasoning_effort`, with native tools,
  simulated function calling, or no tools.
- Keep `none` as the fallback for the models that require it with native
  tools, except when the caller enabled reasoning without choosing an
  effort, in which case the model default applies.
- Only `effort` maps onto Chat Completions; `summary` and `exclude` are
  not forwarded.
- Type `reasoning_effort` on the request as `ChatCompletionReasoningEffort`.
- Cover the resolver and adapter with tests for the forwarding and
  fallback behaviour.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…tor adapter

Forwarding the caller's reasoning effort from the OpenAI connector adapter
let an explicit effort override `reasoning_effort: 'none'` on gpt-5.6,
gpt-6-luna and gpt-6-sol with native tools, which OpenAI rejects. It also
sent `reasoning_effort` to models that do not accept the parameter, turning
requests that worked before into a 400.

`none` is a constraint of those models with native tools, not a default,
and forwarding a caller effort safely needs to know which models accept
`reasoning_effort`, which the connector has no source for.

- Stop forwarding the caller's `reasoning` from the OpenAI connector
  adapter, as before the forwarding was added.
- Always send `reasoning_effort: 'none'` for gpt-5.6, gpt-6-luna and
  gpt-6-sol with native tools, and log at debug level when that overrides
  reasoning the caller asked for.
- Restore the `reasoning_effort` request type to the SDK type plus `none`.
- Replace the forwarding tests with tests for the override, the log and
  the absence of forwarding for other models.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants