Skip to content

fix(reflect): enforce max_tokens on the done-tool completion path (#2756) - #2775

Closed
benfrank241 wants to merge 1 commit into
mainfrom
fix/reflect-done-tool-max-tokens
Closed

fix(reflect): enforce max_tokens on the done-tool completion path (#2756)#2775
benfrank241 wants to merge 1 commit into
mainfrom
fix/reflect-done-tool-max-tokens

Conversation

@benfrank241

Copy link
Copy Markdown
Member

Fixes #2756.

Problem

A mental model's max_tokens is forwarded through refresh_mental_modelreflect_asyncrun_reflect_agent (the #1076 fix), and the agent enforced it on its forced-final path (max_completion_tokens) and its direct-text short-circuit path (rewrite when count_cl100k_tokens(answer) > max_tokens). But the done-tool path — how the agent normally finishes — took the answer verbatim from the tool-call arguments (_process_done_tool) with no token check and no rewrite.

Those tool-call arguments come from call_with_tools, which is intentionally uncapped to leave headroom for tool-call JSON. So the cap was only enforced on the two paths the agent rarely takes, and mental-model content routinely overshot max_tokens by several times.

Agent completion path max_tokens enforced (before)
Forced final (max iterations)
Direct-text short-circuit
done tool (normal path)

Verified against the reporter's repro (max_tokens=256 → ~1,150-token content; max_tokens=800 → ~1,360 tokens, well-formed, not truncated → the cap was simply never applied).

Fix

Extract the short-circuit path's capped-rewrite logic into a shared _enforce_answer_token_cap helper and route all three completion paths through it, so the cap is honoured no matter how the agent finishes. The done-tool path now:

  • runs the same single capped rewrite when the answer overshoots,
  • folds the rewrite's token usage into its accounting,
  • adds a final_rewrite LLM-trace entry (matching the short-circuit path).

The rewrite happens before structured-output generation so structured output is derived from the capped answer. No behaviour change when the answer is already within budget (no extra call) or when no cap is configured.

Tests

Deterministic regression tests over _process_done_tool:

  • over-budget answer → exactly one capped rewrite (max_completion_tokens forwarded, scope="reflect", usage folded in, final_rewrite traced), rewritten text returned;
  • under-budget answer → no rewrite call, answer untouched;
  • max_tokens=None → no-op;
  • helper-level no-cap guard.

59 deterministic tests in test_reflect_agent.py pass (real-LLM judge tests excluded from the local run).

Reported by @sphynx79 with a clean root-cause trace and repro.

)

A mental model's max_tokens is forwarded to the refresh reflect call and the
agent capped its forced-final and direct-text short-circuit paths — but the
done-tool path, which is how the agent normally finishes, took the answer
verbatim from the (intentionally uncapped) tool-call arguments with no token
check. So the cap was only enforced on the two paths the agent rarely takes,
and mental-model content routinely overshot max_tokens by several times.

Extract the short-circuit path's capped-rewrite logic into a shared
`_enforce_answer_token_cap` helper and route all three completion paths
through it, so the cap is honoured no matter how the agent finishes. The
done-tool path now folds the rewrite's token usage into its accounting and
adds a `final_rewrite` trace entry, matching the short-circuit path.

Tests: direct regression tests over `_process_done_tool` — an over-budget
answer triggers exactly one capped rewrite (budget forwarded, usage folded
in, traced), an under-budget answer makes no call, and max_tokens=None is a
no-op.
@nicoloboschi

Copy link
Copy Markdown
Collaborator

superseded by #2757

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] mental model max_tokens not enforced when reflect agent finishes via the done tool (the common path)

2 participants