harness: dependency timeout during failure finalization can leave turn running after reconnect

[McHersheys]

Current verification status — 2026-09-06

The original incident report below is retained as historical evidence. Subsequent work now separates two branches:

1. Failed-turn finalization: 7/7 controlled unchanged-release dependency fault/control cases reproduced the swallowed read/write-failure path and distinguished a committed write with a lost acknowledgement from an absent write.
2. Later stranded admission: a separate real Engine/Queue handoff plus one deliberately lost reply reproduced a failed predecessor, new Running step0 successor and two in-flight deliveries. The diagnostic recovered at its shortened2.5s queue deadline. The initiating live outage and an indefinite wedge are not proven by that result.

Executed fixtures, full branch matrix and limitations.

Current main c86b4ddbd414eb43683cc70c6a3245b7e56c0cc3 already propagates failure-finalization errors and holds the session lock. A separate narrow follow-up patch confirms a timed-out terminal write by one exact whole-record readback without repeating work. It has a genuine failing regression baseline,4/4 passing regressions,455/455 passing library tests, formatting and Clippy checks. Tested patch and precise scope.

That patch is not deployed or accepted upstream; PR submission awaits the required matching MOT ticket. The later queue branch and original live initiating fault remain separate. No uncertain tool effects were replayed; final deployment and autonomous acceptance remain blocked.

---

Original incident report — preserved

Observed failure

An unmodified released Harness worker left a turn durably running after logging that it was finalising that turn as failed. Its SDK subsequently reconnected to the Engine, but the turn did not reach a terminal state or continue autonomously. harness::stop acknowledged stopping: true; readback showed the abort request but the same running step.

This is an observed integration fault plus a source-level hypothesis, not yet a minimal deterministic reproducer or a verified fix.

Version and scope

• Harness release harness-v1.8.13, release source deb6f98dbd3c7cbc226234844761430a47b911cf.
• Unmodified worker binary SHA-256: 563f96d9b59fa1a9b93c3b9b1e16aeda4aee5f601a247a622c93bbc5f28a0b03.
• Linux amd64, bounded gVisor sandbox, private local Engine connection.
• The relevant fail_turn error-handling shape is also present in the inspected 1.8.15 source and current main (00417f34dcee7465d5d4602f4ab497db106f3e55). This does not assert the complete live reproduction was repeated on those versions.

Sanitized timeline (2026-09-06 UTC)

08:54:48.725894  turn started
08:55:05.722388  ERROR turn step failed; finalising turn as failed
                       error=harness/dependency: session::messages: invocation timed out
08:55:13.556042  WARN expiry sweep skipped: binding store unreadable
                       error=harness/state: harness::state::list harness_binding: invocation timed out
08:55:53.558216  WARN expiry sweep skipped: binding store unreadable (same timeout)
08:55:55.722347  WARN no frames from engine within idle window; forcing reconnect
                       idle_timeout=60s
08:56:03.540050  INFO iii connected

After reconnect the same turn remained at running step 1. Readback found no pending function calls or child turns. The only observed work-tool action was read-only function discovery. An ordinary provider request had completed successfully; this is not evidence of a model-generation hang.

A subsequent explicit stop was acknowledged and recorded, but the turn remained running. A single explicit reconciliation wake also failed to terminalize it. Only a later operator-controlled cold restart cancelled/drained the old turn. No uncertain external operation was automatically repeated. That recovery is not a reliable product fix.

Source-level hypothesis

In harness/src/turn_loop.rs, fail_turn converts a state-read error to absence:

let record = crate::state::get_turn(...).await.ok().flatten();

It then converts a finalization failure to skipped(session_id):

finalize_failed(...).await.unwrap_or_else(|_| skipped(session_id))

skipped reports Running, next_step: None, skipped: true. Thus a dependency failure while trying to finalize can become indistinguishable to the caller from a benign no-op, without an established future terminalization step. The observed outage overlaps this path, but fault injection is still needed to prove the exact failure cut and the Engine/SDK cause of the original outage.

Expected behavior / useful regression coverage

• Dependency unavailability during failure finalization should remain an explicit, recoverable failure to terminalize, not a silent running/no-next-step result.
• Test disconnect/read timeout at get_turn, session finalization, and durable turn-write cuts; restore transport and verify an eventual truthful terminal observation through the owning lifecycle contract.
• Test stop while failure finalization is unavailable, plus reconnect/re-registration and cold restart.
• Preserve exact turn identity and cancellation ordering. Never replay an outcome-unknown external tool action merely to complete a turn.
• Distinguish failed state persistence from a stale/terminal turn legitimately skipped.

No downstream Harness/Engine fork or restart watchdog has been introduced. This remains a reliability blocker for autonomous work through the affected runtime.

[McHersheys]

Deterministic diagnostics: two distinct branches, not one failure

The original report's “not yet a minimal deterministic reproducer” is now superseded for the failure cuts below. These are fault-injection results against the unchanged released Harness binary, not a rewritten implementation or a deployed fix. The initiating Engine/SDK outage in the original integration incident is still unproven.

Exact exercised artifacts / safety

• Release source: deb6f98dbd3c7cbc226234844761430a47b911cf, Harness 1.8.13.
• Harness SHA-256: 563f96d9b59fa1a9b93c3b9b1e16aeda4aee5f601a247a622c93bbc5f28a0b03.
• Engine SHA-256: 988bd72aba10781d3c3bce5aab4bce96471273fe5444ec968ae813d3fefc186b.
• Standalone Queue SHA-256: 6f54e9fca9002ac296a275957d6b70198b2dad942327677bbdd325afa951f15b (second experiment).
• Fresh networkless Linux amd64 gVisor containers, 2 CPU / 2 GiB; no credentials, channels, model requests, or work-tool effects. State/configuration/session RPCs are explicitly labelled test doubles. The Engine, Harness, and (in branch B) Queue are real released executables.
• Harness session RPC timeout shortened to 300 ms; the session-message test double returns only after 1.2 s. Thus timeout is real, not an invented error string. Exact turn state, writes, queue snapshot, and wire replies were inspected.
• No product sources changed; no restart watchdog, fork, or operator recovery was counted as acceptance. Diagnostic container/source cleanup verified.

A. Failure finalization can silently acknowledge nonterminal state — confirmed

Seed a running turn at step 1 / turn_count 1 with no calls or children. Invoke actual harness::turn; make session::messages time out. Independently vary ONLY the finalization dependency:

Fault/control	Durable state	Actual turn RPC result
State read/write available	failed	failed, skipped=false
get_turn during fail_turn times out	running, unchanged	running, skipped=true, no next_step
Failed-record write explicitly rejected	running, unchanged	running, skipped=true, no next_step
Both failed-record writes time out without committing	running, unchanged	running, skipped=true, no next_step
Both failed-record writes commit but acknowledgements are lost	failed	running, skipped=true, no next_step
Session error append / error status update reject AFTER durable failed write	failed	failed, skipped=false

All six controls/fault cases produced those outcomes. This narrows the boundary: fail_turn's .ok().flatten() / unwrap_or_else(...skipped...) masks unavailable finalization; a failure of the later best-effort session presentation alone did not strand the durable turn.

The lost-ack control matters: two writes carried identical failed-record bytes. A running/skipped RPC reply is not proof that finalization did not commit, and must never authorize repeating an uncertain external action.

A seventh cut seeded a genuinely new message into harness_queue during the timeout, allowed predecessor finalization, but omitted the Engine queue provider. Harness persisted the old turn failed, drained the new message, persisted a new turn running/step0, exhausted its five enqueue attempts, logged reseed after finalize drain failed; a parked notification may be stranded, then returned the old turn's failed result. This is a separate seed-to-enqueue gap, not the same failed-finalization case. queue::define was an explicit test double in this negative cut, not evidence of a healthy real queue.

B. Failed predecessor + new step0 + two in-flight deliveries — separately reproduced

A later integration incident must not be conflated with A:

14:35:12.592Z  old turn errors: context::count-tokens invocation timed out
14:35:12.629Z  old turn emits completed/failed, terminal=true
               a new turn is durably running, step=0, turn_count=0
               no new turn-started event, no calls/children
               queue store: old step4 + new step0 in-flight; queued/DLQ empty

That old turn did finalize. The later state alone does not establish an infinite wedge, nor which transport event initiated the stall.

I built a second deterministic fixture with the real standalone Queue, file-backed builtin adapter, grouped FIFO (message_group_field=session_id), and an otherwise transparent WebSocket proxy on the Harness connection:

1. Submit old step1 through actual TriggerAction::Enqueue.
2. While session::messages is timing out, inject one genuine fixture user message into the Harness pending-message store.
3. Harness commits old turn failed and seeds/enqueues the new step0.
4. Drop exactly one outgoing invocationresult for the old harness::turn. Keep both sockets alive; forward all other frames; never replay the dropped result.
5. Inspect before the queue's invocation deadline.

Observed: old failed commit exists; new turn is running/step0/turn_count0; no new message read/turn-started yet. The actual file-backed queue snapshot has exactly two in-flight entries for that session (old step1 and new step0), queues={}, dlqs={}. Bare topic stats misleadingly showed depth=0 and consumer_count=0, so those stats were not treated as proof of no work.

Control: with no dropped reply, the successor ran immediately; both old and new turns reached failed via the intentionally failing dependency, in ~0.75 s total.

Deadline control: shorten ONLY this diagnostic queue's timeout_ms to 2500 and backoff to 100 ms. The two-in-flight state resolves at the configured deadline: Queue retries the old delivery, Harness's exact stale-turn guard skips it without reading messages again, then the new turn runs. Total ~2.99 s; exactly one message read for each of the two turns; zero work-tool effects. Repeated in two fresh containers with the same outcome.

This identifies a concrete reply/queue group-lock boundary that can produce the later live signature. Queue's documented default target deadline is 1,800,000 ms (30 minutes). The live incident was operator-recovered before that deadline, so we cannot claim it would never recover, or that its reply was definitely lost. The original connection/reconnect fault still needs tracing at the Engine/SDK boundary. Simply lowering production deadlines could cause unsafe retries and is not proposed as a fix.

Repair / regression direction (proposal, not applied)

• Give failed finalization an explicit durable recovery path / failure outcome distinct from benign stale delivery. Recovery must be terminalization-only and reconcile a possibly committed write, not rerun model/tool execution.
• Exercise absent commit and committed/lost-ack independently, exact turn identity, stop races, and stale successor fencing.
• Cover the new-record-to-enqueue gap separately; do not hide exhausted reseed admission behind the predecessor's terminal result.
• Expose actual in-flight/group-wait reason and deadline. Investigate why a completed Harness invocation's result is absent after the live reconnect; do not bypass the FIFO fence to make progress look healthy.
• Preserve the safe stale-turn control: a known stale delivery must not regenerate or redispatch tools. No outcome-unknown tool call was replayed in these diagnostics.

These results keep autonomous deployment blocked; they do not establish a repaired runtime. Happy to split the queue/admission branch into its own issue if that fits ownership better.

Reproducer wiring

The exact executed TypeScript fixtures below register the stated test doubles and drive real RPCs; wire_proxy.py is the executed transparent drop gate. They require Deno with iii-sdk, Python with websockets, and the exact executables above. In an isolated workspace create /tmp/diag, start an Engine on loopback 49134 with iii-worker-manager and iii-stream, start the selected fixture, wait for dependencies-ready, then start Harness. For B, start Queue before Harness and connect Harness through the proxy on 49135. Do not run against an existing user's Engine/state. The tests write no external repository/channel/tool effects.

The orchestration is a test envelope, not a deployment recipe. A pair of earlier proxy attempts used the wrong wire spelling (invocation_result rather than the actual lowercase invocationresult); those attempts did not inject a fault and were rejected. The successful results below use the verified wire discriminator.

harness_fixture.ts — SHA-256 f991df6a10032d41fc8f2ff3c50063f81318a1d2165c065b3a56004588df1115

// Diagnostic only: real released Harness + Engine, explicit dependency test doubles.
// No provider/tool function is registered and the container has no network.
import {registerWorker} from 'iii-sdk';
const iii=registerWorker('ws://127.0.0.1:49134',{workerName:'harness-diagnostic-fixture'});
const delay=(ms:number)=>new Promise(r=>setTimeout(r,ms));
const clone=(x:any)=>JSON.parse(JSON.stringify(x));
const cases=new Map<string,any>();const store=new Map<string,any>();const trace:any[]=[];
const cfg={session_timeout_ms:300,context_timeout_ms:300,dispatch_timeout_ms:1000,default_functions:null,default_filesystem_root:'off'};
const note=(fn:string,p:any)=>trace.push({n:trace.length,fn,p:clone(p)});
const reg=(id:string,fn:(p:any)=>any)=>iii.registerFunction(id,async(p:any)=>{note(id,p);return await fn(p);});
const key=(p:any)=>p.scope+'/'+p.key;
const get=async(p:any)=>{const c=cases.get(p.key);if(p.scope==='harness_turn'&&c?.failed&&c.mode==='read-timeout'){await delay(1200);throw Error('fixture read timeout');}return clone(store.get(key(p))??null);};
const put=async(p:any)=>{const c=cases.get(p.key);if(p.scope==='harness_turn'&&p.value.status==='failed'){
 if(c?.mode==='write-reject')throw Error('fixture rejected terminal write');
 if(c?.mode==='write-timeout'){await delay(1200);throw Error('fixture dropped terminal write');}
 if(c?.mode==='write-lost-ack'){store.set(key(p),clone(p.value));c.writes.push(clone(p.value));await delay(1200);return {};}
 }store.set(key(p),clone(p.value));if(c&&p.scope==='harness_turn')c.writes.push(clone(p.value));return {};};
reg('state::get',get);reg('state::set',put);reg('state::delete',(p)=>{store.delete(key(p));return {};});
reg('state::list',(p)=>[...store.entries()].filter(([k])=>k.startsWith(p.scope+'/')).map(([,v])=>clone(v)));
reg('harness::state::get',get);reg('harness::state::list',()=>[]);reg('harness::state::compare-and-set',()=>({swapped:true}));reg('state::claim-namespace',()=>({}));
reg('configuration::get',()=>({value:cfg}));reg('configuration::register',()=>({}));reg('configuration::set',()=>({}));
reg('directory::skills::list',()=>({skills:[]}));reg('queue::define',()=>({defined:true}));
reg('session::get',(p)=>({meta:{session_id:p.session_id,title:'Isolated diagnostic',metadata:{}}}));
reg('session::ensure',(p)=>({meta:{session_id:p.session_id,metadata:{}},created:false}));
reg('session::set-status',(p)=>{const c=cases.get(p.session_id);if(c?.mode==='post-write-errors'&&p.status==='error')throw Error('fixture status rejected');return {};});
reg('session::append',(p)=>{const c=cases.get(p.session_id);if(c?.mode==='post-write-errors')throw Error('fixture append rejected');return {entry_id:p.entry_id??'e_fixture'};});
reg('session::messages',async(p)=>{const c=cases.get(p.session_id);if(!c)throw Error('Unknown fixture session');c.failed=true;
 if(c.mode==='reseed')store.set('harness_queue/'+p.session_id+':q1',{id:'q1',session_id:p.session_id,entry_id:'e_q1',queued_at:1,message:{role:'user',content:[{type:'text',text:'new isolated message'}],timestamp:1}});
 await delay(1200);return {entries:[]};
});
const rows:any[]=[];
function check(ok:boolean,message:string){if(!ok)throw Error(message);}
async function ready(id:string){const end=Date.now()+15000;for(;;){try{await iii.trigger({function_id:'engine::functions::info',payload:{function_id:id},timeoutMs:1000});return;}catch(e){if(Date.now()>end)throw e;await delay(100);}}}
try{
 await ready('session::messages');await Deno.writeTextFile('/tmp/diag/dependencies-ready','ready');await ready('harness::turn');
 for(const mode of ['healthy-finalization','read-timeout','write-reject','write-timeout','write-lost-ack','post-write-errors','reseed']){
  const sid='s_fixture_'+mode,tid='t_fixture_'+mode;const c={mode,failed:false,writes:[] as any[]};cases.set(sid,c);
  const record={session_id:sid,turn_id:tid,status:'running',step:1,turn_count:1,depth:0,abort:false,calls:{},options:{model:'gpt-6-astra',provider:'openai',max_turns:3,functions:null,max_transient_resumes:0},created_at:1,updated_at:1};store.set('harness_turn/'+sid,clone(record));
  const start=Date.now(),traceStart=trace.length;let result:any,error:string|undefined;
  try{result=await iii.trigger({function_id:'harness::turn',payload:{session_id:sid,turn_id:tid,step:1,depth:0},timeoutMs:10000});}catch(e){error=String(e);}
  const durable=clone(store.get('harness_turn/'+sid));
  let valid=true,assertionError:string|undefined;
  try{
   check(c.failed,'session::messages timeout not reached');check(!error,'turn RPC failed: '+error);
   if(['healthy-finalization','post-write-errors'].includes(mode)){check(durable.status==='failed'&&result.status==='failed'&&!result.skipped,'normal durable failure not finalized');}
   if(['read-timeout','write-reject','write-timeout'].includes(mode)){check(durable.status==='running'&&result.status==='running'&&result.skipped&&result.next_step==null,'expected stranded finalization not observed');}
   if(mode==='write-lost-ack'){check(durable.status==='failed'&&result.status==='running'&&result.skipped,'lost-ack control mismatch');check(c.writes.length===2&&JSON.stringify(c.writes[0])===JSON.stringify(c.writes[1]),'terminal retry payload not identical');}
   if(mode==='reseed'){check(c.writes.some((v:any)=>v.turn_id===tid&&v.status==='failed'),'predecessor failed commit missing');check(durable.status==='running'&&durable.turn_id!==tid&&durable.step===0,'new admission not seeded');}
   check(Object.keys(durable.calls).length===0,'external call appeared');
  }catch(e){valid=false;assertionError=String(e);}
  const row={case:mode,seconds:(Date.now()-start)/1000,result,error,durable,writes:c.writes,valid,assertionError,trace:trace.slice(traceStart)};rows.push(row);console.log('CASE '+JSON.stringify({...row,trace:undefined,writes:c.writes.map(v=>({status:v.status,turn_id:v.turn_id,step:v.step}))}));
 }
 await delay(1400);await Deno.writeTextFile('/tmp/diag/result.json',JSON.stringify({kind:'explicit-dependency-fault-injection',rows,trace},null,2));
 check(rows.every(r=>r.valid),'one or more diagnostics did not meet their stated expectations');
}finally{await iii.shutdown();}

harness_queue_fixture.ts — SHA-256 43b86186ac53706c55cb829377f57a1a58904a2a9ee589fae485372841c69f99

// Queue diagnostic: unchanged Harness/Engine/Queue, fixture state/session dependencies.
import {registerWorker} from 'iii-sdk';
const iii=registerWorker('ws://127.0.0.1:49134',{workerName:'harness-queue-diagnostic'});
const delay=(ms:number)=>new Promise(r=>setTimeout(r,ms));const clone=(x:any)=>JSON.parse(JSON.stringify(x));
const store=new Map<string,any>(),cases=new Map<string,any>();const trace:any[]=[];let config:any={adapter:{name:'builtin',config:{store_method:'file_based',file_path:'/tmp/diag/queue'}},queue_configs:{}};
const cfg={session_timeout_ms:300,context_timeout_ms:300,dispatch_timeout_ms:1000,default_functions:null,default_filesystem_root:'off'};
const reg=(id:string,fn:(p:any)=>any)=>iii.registerFunction(id,async(p:any)=>{trace.push({fn:id,p:clone(p),at:Date.now()});return await fn(p);});
const get=(p:any)=>clone(store.get(p.scope+'/'+p.key)??null);
reg('state::get',get);reg('state::set',p=>{store.set(p.scope+'/'+p.key,clone(p.value));if(p.scope==='harness_turn')cases.get(p.key)?.writes.push(clone(p.value));return {};});
reg('state::delete',p=>{store.delete(p.scope+'/'+p.key);return {};});reg('state::list',p=>[...store.entries()].filter(([k])=>k.startsWith(p.scope+'/')).map(([,v])=>clone(v)));
reg('harness::state::get',get);reg('harness::state::list',()=>[]);reg('harness::state::compare-and-set',()=>({swapped:true}));reg('state::claim-namespace',()=>({}));
reg('configuration::get',p=>({value:p.id==='queue'?config:cfg}));reg('configuration::register',()=>({}));reg('configuration::set',p=>{if(p.id==='queue')config=clone(p.value);return {value:config};});
reg('directory::skills::list',()=>({skills:[]}));
reg('session::get',p=>({meta:{session_id:p.session_id,title:'Queue diagnostic',metadata:{}}}));reg('session::set-status',()=>({}));reg('session::append',p=>({entry_id:p.entry_id??'e_fixture'}));
reg('session::messages',async p=>{const c=cases.get(p.session_id);if(!c)throw Error('Unknown fixture');const r=store.get('harness_turn/'+p.session_id);c.messages.push(r.turn_id);
 if(r.turn_id===c.old)store.set('harness_queue/'+p.session_id+':q1',{id:'q1',session_id:p.session_id,entry_id:'e_q1',queued_at:1,message:{role:'user',content:[{type:'text',text:'new input, no external effects'}],timestamp:1}});
 await delay(1200);return {entries:[]};});
function check(v:any,s:string){if(!v)throw Error(s);}
async function until(f:()=>any,ms=15000){const end=Date.now()+ms;while(!await f()){if(Date.now()>end)throw Error('bounded wait expired');await delay(40);}}
async function ready(id:string){await until(async()=>{try{await iii.trigger({function_id:'engine::functions::info',payload:{function_id:id},timeoutMs:500});return true;}catch{return false;}});}
const rows:any[]=[];
try{
 await ready('session::messages');await Deno.writeTextFile('/tmp/diag/dependencies-ready','ready');await ready('queue::define');await ready('harness::turn');
 // Let Harness finish its own default define first; then shorten only this diagnostic queue's deadline.
 await until(async()=>{try{return(await Deno.readTextFile('/tmp/diag/harness.log')).includes('harness ready:');}catch{return false;}});
 await iii.trigger({function_id:'queue::define',payload:{queue:'harness-turn',config:{type:'fifo',message_group_field:'session_id',concurrency:10,max_retries:1,backoff_ms:100,poll_interval_ms:10,timeout_ms:2500,redeliver_on_engine_restart:true}},timeoutMs:5000});
 for(const mode of ['healthy-handoff','lost-result']){
  const sid='s_fixture_'+mode,old='t_fixture_'+mode;const c={old,writes:[] as any[],messages:[] as string[]};cases.set(sid,c);
  store.set('harness_turn/'+sid,{session_id:sid,turn_id:old,status:'running',step:1,turn_count:1,depth:0,abort:false,calls:{},options:{model:'gpt-6-astra',provider:'openai',max_turns:3,functions:null,max_transient_resumes:0},created_at:1,updated_at:1});
  await Deno.writeTextFile('/tmp/diag/fault.json',JSON.stringify({drop_session:mode==='lost-result'?sid:null}));
  const start=Date.now();const admission=await iii.trigger({function_id:'harness::turn',payload:{session_id:sid,turn_id:old,step:1,depth:0},action:{type:'enqueue',queue:'harness-turn'},timeoutMs:5000});
  await until(()=>c.writes.some(v=>v.turn_id!==old),6000);
  let snapshot:any,stats:any;
  if(mode==='lost-result'){
   await until(async()=>{try{return (await Deno.readTextFile('/tmp/diag/proxy.jsonl')).includes('dropped');}catch{return false;}},3000);
   await delay(150);snapshot=clone(store.get('harness_turn/'+sid));stats=await iii.trigger({function_id:'engine::queue::topic_stats',payload:{topic:'harness-turn'},timeoutMs:1000});
   await Deno.writeTextFile('/tmp/diag/queue-snapshot-request','read');await until(async()=>{try{await Deno.stat('/tmp/diag/queue-snapshot.json');return true;}catch{return false;}},2000);
  }
  let recoveryError:string|undefined;try{await until(()=>store.get('harness_turn/'+sid)?.status==='failed'&&store.get('harness_turn/'+sid)?.turn_id!==old,6500);}catch(e){recoveryError=String(e);}
  const final=clone(store.get('harness_turn/'+sid));let valid=true,error:string|undefined;
  try{check(c.writes.some(v=>v.turn_id===old&&v.status==='failed'),'predecessor terminal commit absent');check(final.turn_id!==old,'successor not seeded');check(!recoveryError,'bounded queue deadline did not recover');check(c.messages.length===2,'old effects retried or successor missing');check(Object.keys(final.calls).length===0,'tool effects not zero');if(mode==='lost-result')check(snapshot.status==='running'&&snapshot.step===0&&snapshot.turn_count===0,'stranded-admission snapshot not observed');}catch(e){valid=false;error=String(e);}
  const row={case:mode,seconds:(Date.now()-start)/1000,admission,snapshot,stats,final,writes:c.writes,messages:c.messages,recoveryError,valid,error};rows.push(row);console.log('CASE '+JSON.stringify(row));
 }
 await Deno.writeTextFile('/tmp/diag/result.json',JSON.stringify({kind:'real-queue-controlled-lost-reply',rows,trace},null,2));check(rows.every(r=>r.valid),'diagnostic assertions failed');
}finally{await iii.shutdown();}

wire_proxy.py — SHA-256 60267688601941bd5341a3cab0d70a57f53ae730e66d5ed19ce11dc56159b1a6

"""Fixture-only transport gate. Forward unchanged; drop one named turn reply, never replay."""
import asyncio,json,pathlib
from websockets.asyncio.server import serve
from websockets.asyncio.client import connect
root=pathlib.Path('/tmp/diag');dropped=set()
def note(v):
 with (root/'proxy.jsonl').open('a') as f:f.write(json.dumps(v)+'\n')
async def handler(client):
 async with connect('ws://127.0.0.1:49134'+client.request.path,max_size=4*1024*1024) as upstream:
  async def pump(src,dst,outbound):
   async for raw in src:
    try:m=json.loads(raw)
    except Exception:m={}
    if outbound and m.get('type')=='invocationresult' and m.get('function_id')=='harness::turn':
     note({'turn_reply':m})
     try:target=json.loads((root/'fault.json').read_text()).get('drop_session')
     except (FileNotFoundError,ValueError):target=None
     sid=(m.get('result') or {}).get('session_id')
     if target and sid==target and sid not in dropped:
      dropped.add(sid);note({'dropped':m});continue
    await dst.send(raw)
  tasks=[asyncio.create_task(pump(client,upstream,True)),asyncio.create_task(pump(upstream,client,False))]
  done,pending=await asyncio.wait(tasks,return_when=asyncio.FIRST_COMPLETED)
  for t in pending:t.cancel()
async def snapshot():
 while True:
  if (root/'queue-snapshot-request').exists() and not(root/'queue-snapshot.json').exists():
   data={str(p.relative_to(root)):p.read_text() for p in (root/'queue').rglob('*') if p.is_file() and p.stat().st_size<4000000}
   (root/'queue-snapshot.json').write_text(json.dumps(data))
  await asyncio.sleep(.025)
async def main():
 async with serve(handler,'127.0.0.1',49135,max_size=4*1024*1024):
  (root/'proxy-ready').write_text('ready');await snapshot()
asyncio.run(main())

[McHersheys]

Current main moved to c86b4ddbd414eb43683cc70c6a3245b7e56c0cc3 during this investigation. That merge already adds the per-session lock and propagates failed-finalization read/write errors instead of swallowing them. I am basing follow-up work on that source, not duplicating the released-binary fix.

The remaining regression boundary is recovery after that error: retry finalization only (reconciling a possibly committed terminal write and exact turn identity), not model/tool execution. The successor-admission / lost predecessor reply remains a separately tracked boundary; the controlled queue deadline result above does not establish the original transport cause.

The repository contribution guide requires a matching iii-team Linear ticket. I do not have access to that Linear workspace. Could a maintainer link or create the appropriate MOT ticket for the follow-up to #1108? I will not invent an identifier or use the ticketless exception for a behavior fix. No downstream product fork or deployment is being made.

[McHersheys]

Current-main follow-up: bounded confirmation of a lost terminal-write acknowledgement

Current main c86b4ddbd414eb43683cc70c6a3245b7e56c0cc3 already propagates errors from fail_turn and takes the session lock (merged in #1102). This narrow follow-up does not duplicate that fix.

A remaining source-level cut is when the failed-turn write commits but both existing write acknowledgements time out. Propagating that timeout exits the current finalizer before failure publication and queued-input continuation, despite the exact failed record already being stored.

The patch below keeps the existing write and its one identical-record retry. Only on a timeout, it performs one bounded readback under the existing session lock. An exact match of the whole TurnRecord allows the same in-flight finalizer to continue. Missing/different/unreadable state retains the original write error. Known success and non-timeout rejection do not read back. It adds no write, generation retry, tool replay, queue deadline change or recovery worker.

Actual component verification

• Regression sensitivity before the behavior change: 2 intended failures, 2 passing controls.
• After the change: 4/4 regressions PASS; 455/455 Harness library tests PASS.
• Exact Rust 1.97.1, frozen Cargo lock; rustfmt --edition 2021 --check on both changed files PASS; cargo clippy --locked --offline -p harness --lib -- -D warnings PASS.
• Source-bound embedded UI assets built once from the same upstream source and frozen pnpm lock (pnpm11.13.1/Node22.23.2), then reused unchanged. Rust compilation/tests/lint ran networkless in isolated gVisor containers. No placeholder UI assets.
• Test/format/lint passes used byte-identical source. Patch apply/reverse checks passed against the stated base/current candidate. Exact test fixtures and the bounded writable cache have been physically removed; existing live candidate/runtime evidence is unchanged.

This is component verification, not deployment or autonomous acceptance. It does not establish the initiating cause of the live reconnect outage, guarantee publication across every crash cut, or fix the separate real-Queue lost-reply/deadline branch documented in the deterministic branch report. No uncertain operation was replayed.

The required matching MOT ticket has been requested above; #1102's MOT-4649 is about split panels/shell UI, so I have not silently assigned this repair to that unrelated ticket. The tested patch is provided here pending the proper association and PR submission.

Applicable patch against c86b4dd

diff --git a/harness/src/state.rs b/harness/src/state.rs
index 32759200..bcc5ad76 100644
--- a/harness/src/state.rs
+++ b/harness/src/state.rs
@@ -176,6 +176,41 @@ pub async fn put_turn(
     }
 }
 
+/// Persist a terminal snapshot under the caller's session lock. A timeout
+/// does not prove the write was absent; reconcile it without replaying a step.
+pub(crate) async fn put_terminal_turn(
+    iii: &IIIClient,
+    record: &TurnRecord,
+    timeout_ms: u64,
+) -> Result<(), HarnessError> {
+    debug_assert!(record.status.is_terminal());
+    confirm_terminal_write(put_turn(iii, record, timeout_ms).await, record, || {
+        get_turn(iii, &record.session_id, timeout_ms)
+    })
+    .await
+}
+
+async fn confirm_terminal_write<T, Read, ReadFuture>(
+    written: Result<(), HarnessError>,
+    expected: &T,
+    mut read: Read,
+) -> Result<(), HarnessError>
+where
+    T: PartialEq,
+    Read: FnMut() -> ReadFuture,
+    ReadFuture: std::future::Future<Output = Result<Option<T>, HarnessError>>,
+{
+    match written {
+        Err(error) if is_timeout(&error) => match read().await {
+            Ok(Some(observed)) if observed == *expected => Ok(()),
+            // Missing, changed, or unreadable state cannot confirm our write.
+            // Preserve the original error rather than retrying any work.
+            _ => Err(error),
+        },
+        other => other,
+    }
+}
+
 /// Whether a state error is the caller-side invocation timeout (the SDK's
 /// wording, carried through [`state_set`]'s error mapping). String-matched
 /// because `HarnessError::State` flattens the SDK error to text.
@@ -500,6 +535,76 @@ pub async fn put_idem(
 mod tests {
     use super::*;
 
+    fn timeout() -> HarnessError {
+        HarnessError::State("state::set: invocation timed out".into())
+    }
+
+    #[tokio::test]
+    async fn terminal_write_lost_ack_is_reconciled_without_another_write() {
+        let expected =
+            json!({"turn_id":"t1","step":3,"status":"failed","calls":{"c1":"triggered"}});
+        let reads = std::cell::Cell::new(0);
+        let result = confirm_terminal_write(Err(timeout()), &expected, || {
+            reads.set(reads.get() + 1);
+            std::future::ready(Ok(Some(expected.clone())))
+        })
+        .await;
+        assert!(
+            result.is_ok(),
+            "the exact terminal snapshot already committed"
+        );
+        assert_eq!(reads.get(), 1);
+    }
+
+    #[tokio::test]
+    async fn terminal_write_unconfirmed_outcomes_remain_errors() {
+        let expected = json!({"turn_id":"t1","step":3,"status":"failed","abort":false});
+        for observed in [
+            None,
+            Some(json!({"turn_id":"t1","step":3,"status":"running","abort":false})),
+            Some(json!({"turn_id":"t2","step":0,"status":"running","abort":false})),
+            Some(json!({"turn_id":"t1","step":3,"status":"cancelled","abort":true})),
+            Some(json!({"turn_id":"t1","step":4,"status":"failed","abort":false})),
+        ] {
+            let reads = std::cell::Cell::new(0);
+            let result = confirm_terminal_write(Err(timeout()), &expected, || {
+                reads.set(reads.get() + 1);
+                std::future::ready(Ok(observed.clone()))
+            })
+            .await;
+            assert!(result.is_err(), "different/missing state is not our commit");
+            assert_eq!(reads.get(), 1);
+        }
+    }
+
+    #[tokio::test]
+    async fn terminal_write_unavailable_readback_keeps_original_error() {
+        let reads = std::cell::Cell::new(0);
+        let result = confirm_terminal_write(Err(timeout()), &json!({"status":"failed"}), || {
+            reads.set(reads.get() + 1);
+            std::future::ready(Err::<Option<Value>, _>(HarnessError::State(
+                "read unavailable".into(),
+            )))
+        })
+        .await;
+        assert!(is_timeout(&result.unwrap_err()));
+        assert_eq!(reads.get(), 1);
+    }
+
+    #[tokio::test]
+    async fn terminal_write_known_success_and_rejection_do_not_read_back() {
+        for written in [Ok(()), Err(HarnessError::State("FORBIDDEN".into()))] {
+            let expected_ok = written.is_ok();
+            let result = confirm_terminal_write(written, &Value::Null, || {
+                panic!("no readback is needed for a known outcome");
+                #[allow(unreachable_code)]
+                std::future::ready(Ok::<Option<Value>, HarnessError>(None))
+            })
+            .await;
+            assert_eq!(result.is_ok(), expected_ok);
+        }
+    }
+
     #[test]
     fn parse_list_handles_array_and_object_shapes() {
         let rec = json!({
diff --git a/harness/src/turn_loop.rs b/harness/src/turn_loop.rs
index 0a1f7307..677ac328 100644
--- a/harness/src/turn_loop.rs
+++ b/harness/src/turn_loop.rs
@@ -2140,7 +2140,7 @@ async fn finalize_failed(
     record.result_error = Some(summary.clone());
     record.updated_at = AgentMessage::now_ms();
     record_failure_telemetry(record, detail, failure);
-    crate::state::put_turn(&deps.iii, record, cfg.session_timeout_ms).await?;
+    crate::state::put_terminal_turn(&deps.iii, record, cfg.session_timeout_ms).await?;
     deps.cancels.clear(&record.turn_id);
     let _ = session
         .append_custom(