Repository navigation
fix(verify): judge the current turn against its tool results - #307
Merged
Merged
Conversation
The verifier side call rendered tool calls as names plus clipped arguments with no results, took the first user message of the whole history as the task, and walked every turn with no size cap. Evidence- based answers therefore failed strict verification by construction and triggered needless hint-mode retries. The trace is now scoped to the latest user turn, pairs each call with its result by tool_call_id, renders a bounded head+tail excerpt per result with explicit omission markers, redacts secrets and wraps every excerpt as untrusted content. The whole trace is capped so the prompt stays bounded regardless of session length, and the verifier is told that excerpts may be truncated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
odek | b04c026 | Commit Preview URL Branch Preview URL |
Oct 08 2026, 07:50 PM |
A follow-up turn that restates what an earlier turn established makes no tool calls, so the current-turn trace alone made strict mode fail correct answers in multi-turn sessions. The verifier now also sees the earlier user prompts and final assistant answers, clamped per message and in total, most recent kept, redacted and wrapped as untrusted content. Earlier turns' tool calls and results still never enter the trace. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
- The turn boundary follows the transcript rule (skip only bg-notice), so a system-initiated wake turn is verified against its own task. - Results pair by position inside a call group with tool_call_id as the override, so providers that omit ids still deliver evidence. - The trace states when earlier calls of the turn were trimmed from context, so their absence is not read as phantom runs. - Secrets are redacted before excerpts are cut, so a key straddling the cut cannot leak as a fragment. - All cuts land on rune boundaries. - The task and earlier user turns use the principal prompt when the surface recorded one, not the resource-expanded text. - The side call renders evidence through the engine's own boundary (no second guard scan) and builds the prompt before the timeout starts. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…the task - Context trimming now counts the current turn's dropped groups separately (index shifted with the slice), so the verifier's trim note cannot excuse phantom tool runs with earlier turns' drops. - The task keeps a bounded excerpt of attached resources next to the typed prompt, so answers given from an attachment stay verifiable. - Tool-call arguments and the answer are redacted before they reach the side call; a result replaced by the trim marker is reported as trimmed rather than sized as output. - Redaction runs on a bounded window, never on a multi-megabyte expansion. - Verifier rendering uses the engine boundary directly; the corrective hint falls back to it when no surface wrapper is installed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
With
verify: {enabled: true, mode: "strict"}, accurate answers built from tool output came backverdict: failand shipped as unverified. Three defects ininternal/loop/verify.go:answer_supersededretries.Fix
tool_call_id.[… N bytes omitted]marker; the stored TOOL RESULT frame is stripped, secrets are redacted, and every excerpt is wrapped as untrusted content via the engine's wrapper or its own nonce'd default. The whole trace is capped at ~14 KB with omitted calls counted.Tests
Verification
go test ./internal/loop/,go test -race ./internal/loop/ -run 'Verify|AnswerSuperseded',go test ./cmd/odek/,ODEK_E2E=true go test ./cmd/odek/ -run TestE2E_all pass.🤖 Generated with Claude Code