Skip to content

fix(verify): judge the current turn against its tool results - #307

Merged
jkyberneees merged 4 commits into
mainfrom
fix/verify-tool-results
Oct 8, 2026
Merged

jkyberneees merged 4 commits into
mainfrom
fix/verify-tool-results

Conversation

@jkyberneees

@jkyberneees jkyberneees commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem

With verify: {enabled: true, mode: "strict"}, accurate answers built from tool output came back verdict: fail and shipped as unverified. Three defects in internal/loop/verify.go:

  1. The trace sent to the verifier contained only tool names and arguments clipped to 160 bytes, never a result. Every claim taken from tool output was unsupported by construction, so strict mode failed nearly every evidence-based answer and hint mode triggered needless answer_superseded retries.
  2. The task was the first user message of the whole history. On serve sessions, from the second prompt on, the answer was judged against an earlier request.
  3. The trace walked every turn with no cap, so the side-call prompt grew with session length.

Fix

  • The trace is scoped to the latest real user turn (bg-notice user messages skipped, matching the engine's existing turn convention) and pairs each call with its result by tool_call_id.
  • Each result is rendered as a head+tail excerpt (~1.5 KB) with an explicit [… N bytes omitted] marker; the stored TOOL RESULT frame is stripped, secrets are redacted, and every excerpt is wrapped as untrusted content via the engine's wrapper or its own nonce'd default. The whole trace is capped at ~14 KB with omitted calls counted.
  • The task is the latest user prompt. Earlier turns reach the verifier only as a bounded block of user prompts and final assistant answers (~1 KB per message, ~4 KB total, most recent kept, redacted, wrapped as untrusted), so a follow-up answer that restates what an earlier turn established is judged against it; earlier turns' tool calls and results never enter the trace.
  • The prompt keeps "prefer fail when in doubt" but tells the verifier excerpts may be truncated: a claim consistent with the visible excerpt counts as supported; contradictions, phantom tool runs and unanswered parts fail. Section labels now match what is sent. Result contents are framed as data, never instructions.
  • No verifier prose reaches clients (unchanged).

Tests

  • Current-turn scoping with results and no leakage from earlier turns; unmatched result handling; per-result and overall-budget truncation markers; untrusted wrapping plus redaction; latest-turn task selection.
  • Regression: the real engine over a two-turn history with a stub verifier that passes only when a figure present solely in the tool output reaches its prompt.

Verification

  • go test ./internal/loop/, go test -race ./internal/loop/ -run 'Verify|AnswerSuperseded', go test ./cmd/odek/, ODEK_E2E=true go test ./cmd/odek/ -run TestE2E_ all pass.
  • Live reproduction with glm-5.3-flash as agent and verifier in strict mode on this branch (4 files, +348/−52): HEAD passed in a single-turn run and on the second turn of a saved session; a control run of the main binary reproduced the bug, failing a correct answer.

🤖 Generated with Claude Code

The verifier side call rendered tool calls as names plus clipped
arguments with no results, took the first user message of the whole
history as the task, and walked every turn with no size cap. Evidence-
based answers therefore failed strict verification by construction and
triggered needless hint-mode retries.

The trace is now scoped to the latest user turn, pairs each call with
its result by tool_call_id, renders a bounded head+tail excerpt per
result with explicit omission markers, redacts secrets and wraps every
excerpt as untrusted content. The whole trace is capped so the prompt
stays bounded regardless of session length, and the verifier is told
that excerpts may be truncated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
odek b04c026 Commit Preview URL

Branch Preview URL
Oct 08 2026, 07:50 PM

jkyberneees and others added 3 commits October 8, 2026 21:22
A follow-up turn that restates what an earlier turn established makes no
tool calls, so the current-turn trace alone made strict mode fail correct
answers in multi-turn sessions. The verifier now also sees the earlier
user prompts and final assistant answers, clamped per message and in
total, most recent kept, redacted and wrapped as untrusted content.
Earlier turns' tool calls and results still never enter the trace.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- The turn boundary follows the transcript rule (skip only bg-notice), so
  a system-initiated wake turn is verified against its own task.
- Results pair by position inside a call group with tool_call_id as the
  override, so providers that omit ids still deliver evidence.
- The trace states when earlier calls of the turn were trimmed from
  context, so their absence is not read as phantom runs.
- Secrets are redacted before excerpts are cut, so a key straddling the
  cut cannot leak as a fragment.
- All cuts land on rune boundaries.
- The task and earlier user turns use the principal prompt when the
  surface recorded one, not the resource-expanded text.
- The side call renders evidence through the engine's own boundary (no
  second guard scan) and builds the prompt before the timeout starts.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the task

- Context trimming now counts the current turn's dropped groups
  separately (index shifted with the slice), so the verifier's trim note
  cannot excuse phantom tool runs with earlier turns' drops.
- The task keeps a bounded excerpt of attached resources next to the
  typed prompt, so answers given from an attachment stay verifiable.
- Tool-call arguments and the answer are redacted before they reach the
  side call; a result replaced by the trim marker is reported as trimmed
  rather than sized as output.
- Redaction runs on a bounded window, never on a multi-megabyte expansion.
- Verifier rendering uses the engine boundary directly; the corrective
  hint falls back to it when no surface wrapper is installed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jkyberneees
jkyberneees merged commit 8123cac into main Oct 8, 2026
10 checks passed
@jkyberneees
jkyberneees deleted the fix/verify-tool-results branch October 8, 2026 19:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant