feat(scripts): evaluate budgeted task resumption with JEV

[ericpaulsen]

Evaluate budgeted task resumption with JEV

Sediment exposes grant-scoped evidence inventory, manifests, and exact reads. The earlier required-lookup comparison recovers a historical constraint but uses more input tokens than full-history injection. Its frozen validator also misses quoted embedded CSV newlines. The next experiment measures whether selecting history before the coding agent starts preserves correctness with less context and lower total observed usage.

Decision: implement a separate, private evaluation driver on main after #85. Reuse the isolated pi controller and shipped evidence routes. Compare no history, full captured history, budgeted keyword selection, and JEV selection. Pin JEV 1.13.0 through the official TypeSafe API for the initial adapter. Credential availability gates live JEV execution. Use synthetic task histories; preserve previous fixtures and results.

Deliverables and dependencies

• T1: Versioned task fixtures and independent validators for required history, unnecessary history, and relevant history mixed with distractors. Include quoted embedded newlines and validator mutation checks. Preserve the original Add agent-requested context retrieval for one previous Session #69 evaluation.
• T2: Consumer-side factual evidence reads, complete-context byte budgeting, deterministic packing, and a bounded JEV adapter. Reuse existing schemas, keyword policy, and HTTP dependency. Validate versioned responses, allowlisted references, and usage. Count every request, candidate prefetch, refusal, and selector token. No automatic retry or hidden fallback.
• T3: Frozen 36-run comparison (three profiles, four arms, three repetitions), isolated source capture and fresh continuations, complete private records, capture/trajectory verification, and truthful usage/cost summaries. T3 depends on T1/T2 contracts; integration depends on both implementations.
• T4: Offline and real API acceptance, live native preflight, one frozen comparison when credentials are available, documentation, independent whole-branch and different-vendor review, and required CI. T4 depends on T1–T3.

Implementation status: 2026-09-23

PR #90 implements T1–T3 at 64a2e9c. All required CI and reviews pass. All 36 continuations finish; the final results report preserves 21 valid original measurements and 15 invalid measurements. Supplemental audits verify reported token usage, capture, and delivery for all 36 Sessions. Final-workspace checks report both-check passes of A 1/9, B 0/9, C 9/9, and D 0/9. JEV provides no historical content and shows no quality-preserving savings.

T4 remains unchecked because the frozen live acceptance remains incomplete. No acceptance amendment or merge is authorized. PR #90 stays draft. Exact original verification traffic counts and bytes remain unknown; the report discloses the premature audit's overlap with two runs. Owned runtime services are stopped after the database archive; private artifacts and the integration branch are retained.

Acceptance

All coding arms use identical model, tools, workspace, visible prompt, and limits. Selection occurs in the experiment consumer at resumption, before the first coding-model request. The history corpus is the final captured call's complete conversation, with preceding-call containment verified. Gold labels and validator files stay outside the coding agent and JEV inputs.

Keyword and JEV arms deliver at most 8,192 UTF-8 bytes of complete serialized historical evidence, including metadata and escaping. Do not describe this as an exact token cap. Model-reported tokens remain the authoritative usage measurement. JEV sees a bounded exact candidate corpus; every prefetch and all JEV input/output usage count even when the coding agent receives none of that content. Source limits, authorization, and Quarantine remain enforced by Sediment.

Record every scheduled result, failed request, and missing usage value. Report correctness and historical constraint recovery alongside input/output/cache usage by model, model/tool/read counts, bytes, latency, and quoted-price estimates with their source/date. Local coding inference has no measured dollar cost unless actual compute metering exists; never present zero API fees as free compute. Fewer coding-model tokens alone do not establish lower total usage or spending.

An unfavorable model result completes an honestly reported experiment; it does not justify tuning and rerunning the held-out comparison. Missing credentials, live isolation, or required checks leave the affected acceptance incomplete and the PR draft. Do not close #69 automatically: this is a separate evaluation with different triggering semantics.

No production API, schema, grant, Fact, Derivation, Attribution, training-label, or database migration changes. No model SDK dependency or generic selector framework. Integration uses one isolated branch; no merge is authorized by this issue.

[ericpaulsen]

Claimed for implementation under the maintainer’s implement-spec request. Integration branch: codex/jev-budgeted-resumption, based on main 6d8c56d. Work uses separate implementer worktrees and preserves the original checkout. JEV credentials gate live evaluation; independent offline work proceeds.

[ericpaulsen]

Budgeted task resumption: retained live results

Execution uses commit 64a2e9c1d6ce449443e1b14ef72f653c21273b67 for PR #90, tracked by issue #89. The frozen comparison contains all 36 scheduled continuations. The original experiment remains incomplete: 21 runs have valid original measurements; 15 retain invalid statuses. Supplemental audits do not replace those verdicts.

The strongest observed result is the keyword baseline: all nine final workspaces pass both the general CSV behavior checks and the historical-constraint checks. JEV supplies no historical evidence in any of its nine continuations. None of its final workspaces passes both checks. This configuration provides no evidence of quality-preserving savings from JEV.

Scope and quality

Three synthetic shipment-CSV profiles each run three repetitions of four arms. The consumer chooses history before the first coding-model request. Each continuation starts a fresh native pi Session. No continuation exposes a retrieval tool to the coding agent, so this experiment does not measure autonomous retrieval during a task.

The coding runtime pins ministral-3:14b-instruct-2512-q4_K_M, pi 0.84.1, Ollama 0.34.2, and LiteLLM 1.100.1. Every arm uses temperature 0, a 16,384-token context setting, a 2,048-token output setting, and a 12-call limit. The source history, workspace, image identities, complete run order, queries, budgets, and thresholds are frozen before the matrix.

The profiles distinguish an omitted historical requirement, a fully stated visible requirement, and relevant history mixed with distractors. Independent checks cover CSV parsing and the case-sensitive historical constraint. The validator and reference labels stay outside coding-agent and selector inputs. C and D receive the same candidate occurrence corpus through Sediment's grant-scoped factual reads.

All 36 retained workspaces receive the same frozen validator in a separate, networkless supplemental audit. The 21 results that the original controller already checked match exactly. The other 15 workspaces receive their first recorded quality checks. None of those 15 passes both checks. The audit makes no model, JEV, or Sediment API calls. It verifies original artifact hashes and workspace identities before and after validation.

Arm	General behavior passes	Historical constraint passes	Both pass
A: no history	7/9	3/9	1/9
B: full history	0/9	9/9	0/9
C: keyword selection	9/9	9/9	9/9
D: JEV selection	6/9	3/9	0/9

Both-check passes by profile, in A/B/C/D order: required history 0/0/3/0; unnecessary history 1/0/3/0; distractors 0/0/3/0. B's workspaces preserve the historical constraint but fail general CSV behavior. Its forced stops prevent treating this as a clean completed-task comparison.

Keyword selection delivers eight complete occurrences in a 7,072-byte envelope for the required and unnecessary profiles. It delivers seven occurrences in an 8,009-byte envelope for distractors. These sizes include serialization and reference metadata and stay below the 8,192-byte cap. Full-history envelopes contain 10,154 or 14,889 bytes. These byte counts are not token counts.

JEV returns raw no_history in all nine requests. The frozen 0.6 confidence threshold produces five insufficient outcomes and four no_history outcomes. Required-history confidences are 0.48, 0.46, and 0.50. Unnecessary-history confidences are 0.80, 0.76, and 0.80. Distractor-history confidences are 0.58, 0.55, and 0.62. Both effective outcomes supply zero historical content. Every request includes the candidate catalog in its input usage; an empty coding prompt addition does not mean zero selection cost.

Original failures and supplemental evidence

Nine full-history runs stop with generation_settings_mismatch. The gate rejects those requests before provider dispatch. Inspection of the pinned pi implementation indicates that its context estimate reduces the output allowance below the frozen 2,048-token setting. The rejected request bodies were not retained, so this field-level explanation is an inference, not an observed wire value. All preceding successful requests remain measured.

Two no-history runs and three JEV runs reach the 12-call limit. Their next attempts stop before provider dispatch. The supplemental accounting audit recognizes known-zero usage only where the frozen gate code, reservation counters, retained records, and terminal reason prove that no provider call occurred. It does not treat arbitrary missing responses as zero.

One JEV run retains capture_read_failed. I caused competing read traffic by requesting a supplemental capture audit while the producer was running. That audit shared the API's single evidence-worker slot from 23:33:43.658 to approximately 23:35:05.465 UTC on 2026-09-22. It overlapped 10-required-D-3 and 11-required-A-3. Capacity competition likely caused the first run's verification failure, but the failed HTTP status was not retained, so the cause remains unconfirmed. The second run's latency is confounded. No task-quality effect is inferred.

The final capture audit runs only after the producer stops. It checks exact Session identity, call count, distinct ordered Fact IDs, the initial prompt, and the complete historical prefix. The gate records have no provider request-ID field for a stronger per-call join. Any successful late verification is separate evidence; it does not change the original capture_read_failed status.

The original comparison.json still reports experiment_complete: false and usage_complete: false. No held-out continuation, prompt, selection threshold, task fixture, or runtime setting changes after outcomes are observed. No failed continuation is replaced by a rerun. Approval to change the acceptance interpretation has not been received.

Supplemental usage accounting

The final audit recovers complete reported input/output usage for all 36 continuations, verifies capture and initial context delivery for all 36, and verifies three source Sessions. It checks fresh, distinct Session identities and preserves all original artifacts. This establishes supplemental accounting and evidence delivery; the original experiment remains incomplete.

Arm	Successful coding calls	Coding input	Coding output	JEV input / output	Combined input + output
A	81	196,543	11,514	0 / 0	208,057
B	93	667,006	40,414	0 / 0	707,420
C	48	221,946	10,608	0 / 0	232,554
D	84	209,076	12,369	52,020 / 2,358	275,823

Coding usage is reported by the pinned local Ministral model; selector usage is reported by JEV 1.13.0. Different tokenizers mean that the arithmetic sum is an accounting total, not equivalent compute. Input totals already include reported cache reads. No coding cache-write count or JEV cache count is reported; those values remain unknown.

Arm	Coding cache reads	Gate attempts	Settings refusals	Call-limit refusals	Native tool calls	Read-first runs
A	180,519	83	0	2	74	9/9
B	625,806	102	9	0	93	0/9
C	205,075	48	0	0	39	9/9
D	192,237	87	0	3	78	9/9

The 14 proven pre-dispatch refusals consume zero provider tokens. The gate's forwarded counter records reservations, not provider dispatches; the audit reconciles that distinction. Every successful provider response contributes its usage. Read-first means that the first native tool invocation is read; it is not evidence of correct task completion.

Arm	Selection evidence HTTP calls	Request bytes	Response bytes	History-envelope bytes per run
A	0	0	0	0
B	36	14,724	160,896	10,154, 14,889
C	45	23,289	227,355	7,072, 8,009
D	36	14,724	160,896	0

JEV makes 9 requests, with 167,115 request bytes and 5,744 response bytes. The selection client counts request-body plus URL-target bytes, and consumed response-body bytes. HTTP headers are excluded. Candidate prefetches count even when nothing enters the coding prompt.

The separate final verification audit makes 712 HTTP calls: 700 with operator authority, 12 with retrieval authority. It records 578,998 request-body bytes, 50,558 URL-target bytes, and 7,401,810 consumed response-body bytes. These are post-run audit traffic, not the original resumption's selection traffic. HTTP headers are excluded.

Original controller verification traffic was not instrumented with the same observer, and inventory polling was not logged. Its exact request counts and byte totals are unknown; the final audit cannot substitute for those measurements. The earlier concurrent audit has 12 recorded retrieval-authority calls and 150 reconstructed operator calls; its interference remains disclosed separately.

Arm	Original elapsed seconds: total	Median	Range	Median selection seconds
A	689.80	75.67	44.28–122.36	0.000
B	2227.01	234.73	213.20–303.95	1.345
C	679.26	58.08	46.79–134.60	1.836
D	732.66	76.20	46.05–124.99	1.720

Elapsed values come from the unchanged original controller. They include its setup and verification work. Two runs overlap the premature audit, and the arms differ in termination and correctness. These values do not support a latency-savings claim.

D reports 18.6% more combined coding-plus-selector tokens than C, while C passes 9/9 tasks and D passes 0/9. Neither this contrast nor lower usage against an unsuccessful full-history arm establishes quality-preserving savings.

The nine JEV calls report 52,020 input and 2,358 output tokens. At the TypeSafe model price recorded in the frozen specification on 2026-09-22 ($0.042 per million input tokens and no listed output charge), the selector-only estimate is $0.00218484. This is a quoted-rate estimate, not measured billing. Local coding compute is not metered, so total dollar cost remains unknown.

Preparation and earlier attempts

Final setup, separate from the 36 continuations, uses 24 native requests with 44,010 input and 973 output tokens, plus one JEV request with 644 input and 64 output tokens. Native cache reads total 37,226 tokens and are already included in input usage.

Retired post-restoration preparation uses 60 native requests with 105,066 input and 2,459 output tokens, plus three JEV requests with 1,932 input and 192 output tokens. The interrupted first matrix has 12 native attempts: 11 complete and one with unknown final usage. Its known subtotal is 30,055 input and 1,888 output tokens. A capture replay adds no inference and is counted once in its original preflight. Earlier artifacts lost during the machine reboot leave pre-restoration usage unknown. These records cannot establish a whole-project token or dollar total.

The retained failed preparations remain visible: an owned gateway lacked the exact local callback origin; the preflight incorrectly rejected a valid JEV abstention; and a delivery verifier accepted only a string although pinned pi sends an exact singleton text array. Those instrumentation fixes and their reviews precede the final frozen matrix. The final verifier accepts only the exact string or exact singleton text array, and still verifies the captured Fact and prefix separately.

Code, review, and CI

The implementation adds evaluation scripts, synthetic fixtures, focused tests, and maintained documentation. Existing factual reads provide the durable evidence. No production Fact, schema, API, grant, Derivation, Attribution, training export, dependency, or legacy evaluation changes.

The maintainer approves exactly the Mako 1.4.3 and Wolfi libexpat1 2.8.5-r0 maintenance entries reviewed in PR #87. Commit 64a2e9c adds those entries without changing prior versions, global expiry, dispositions, or gate predicates. Issue #88 remains separate.

All required workflows pass at that commit: main CI, security, shims, and consumer compatibility. Main CI reports 6,203 passed and 20 skipped tests. Release rehearsal and all four native platforms pass. Security reports all ten jobs and all eight inventory gates passing. All four pinned consumers pass.

The focused evaluation suite passes 104 tests. Security-focused checks pass 35 tests. Real Sediment API acceptance passes 21 checks. Independent whole-branch review and read-only Claude Opus 5 reviews have no unresolved actionable findings. Accepted review findings removed prompt-distorting fixture headers, added a shared catalog bound, and restricted runtime metadata. The later exact text-array verification fix received its own focused independent and Claude reviews. A low-priority request to retain oversize-rejection diagnostics was declined because it does not affect successful execution or acceptance.

Interpretation and acceptance

The keyword result supports a limited claim: bounded evidence from one prior Session can help this fresh agent recover a historical rule and complete these synthetic tasks. JEV's transport and response contract work, but this frozen choice formulation does not select the needed history. Three task profiles and three repetitions do not establish population-level quality or general savings.

T1–T3 are implemented. T4 has passing code checks and reviews, but the frozen live acceptance remains incomplete. PR #90 stays draft under the specification and implement-spec completion gate. Issue #69 remains open and its earlier results remain unchanged. No merge occurs.