fix(sandbox): verify a port-forward is alive before reusing it, and close the env gitignore gap - #1
Conversation
…lose the env gitignore gap kubectl port-forward reports per-connection faults on stderr and keeps running, so a dead tunnel is indistinguishable from a healthy one by poll() alone. The module documented exactly this at _terminate_handle -- "a tunnel can be alive and useless, which looks identical to a healthy one from the outside" -- and then never acted on it. Measured across recorded runs: it hit 4 of ~60. In the worst, forwards opened at 12:11:25 and the engines behind them were unreachable by 12:14:49, processes still alive, nothing reopening. The reports those runs produced blamed the analysis engine rather than the transport. Two halves, and neither works alone. open() now probes an existing tunnel before handing it back. It previously returned early on "same pod + process alive" while its own docstring promised to block until it forwards, so a stage calling prepare_* again to repair a dead tunnel got an immediate return and the same dead tunnel. The probe is the existing HTTP readiness check, which is already the right test -- a TCP connect only proves the port is open. It runs with a single attempt: opening a tunnel means waiting for one to come up and keeps the full fifteen, but checking an existing one is a liveness question whose first failure is the answer. A failed probe is logged as its own reason, distinct from a replaced pod or an exited process, so it reads as the fix firing rather than as a regression. triage_recon and packer_analyst gain prepare_sandbox. The tunnel is opened twice per run, both at intake, and four agents consume those two servers minutes apart; two of them carried no prepare tool and so depended on a tunnel another agent had opened, with no way to repair it. LESSONS_LEARNED #6 already said each analysis agent must prepare its own engine. It had been applied to the two agents someone noticed. A test now asserts the property across all four consumers rather than the two, which is the shape of the original bug: a rule applied case by case leaves holes that stay invisible until a transport flakes. Separately, .gitignore had `env/` -- the virtualenv directory pattern -- but nothing matching a FILE named `env`. One turned up in a working tree holding live provider keys, a single `git add -A` away from this now-public repository. The bare name, the dotted forms and the suffixed forms are all covered now, with .env.example negated back in because it is the template and must stay tracked.
Live validation complete — all three fixes provenPort-forward liveness (this PR). Tested against a real tunnel, not a stubbed one. A
ILSpy sanitization. Live run: 14 of 14 ILSpy responses framed by the membrane across 8 tools, against 0 of 17 before. Also driven offline through the real Critic laundering rules. Full-run signals — all zeroReport written to disk ( Two notes for the recordPeak context reached 228k on
|
Why an MCP engine goes unreachable
kubectl port-forwardreports per-connection faults on stderr and keepsrunning. The tunnel dies; the process does not. From outside, a dead tunnel is
indistinguishable from a healthy one.
The module already documented this at
_terminate_handle— "a tunnel can bealive and useless, which looks identical to a healthy one from the outside" —
and never acted on it.
Measured: 4 of ~60 recorded runs. In the worst, forwards opened at
12:11:25 and the engines were unreachable by 12:14:49 — processes still
alive, nothing reopening. Those runs produced reports blaming the analysis engine
rather than the transport.
Two halves, neither works alone
1.
open()now probes before reusing. It previously returned early onsame pod + process alive, while its own docstring promised to "block until itforwards":
So a stage calling
prepare_*again to repair a dead tunnel got an immediatereturnand the same dead tunnel. The probe is the existing HTTP readinesscheck — already the right test, since a TCP connect only proves the port is open.
It runs with a single attempt: opening waits for a tunnel to come up and keeps
the full 15, but checking an existing one is a liveness question whose first
failure is the answer.
A failed probe logs its own reason (
probe failed), distinct frompod replacedand
process exited— in a live log it reads as the fix firing, not aregression.
2.
triage_reconandpacker_analystgainprepare_sandbox. The tunnel isopened twice per run, both at intake, and four agents consume those two servers
minutes apart:
triage_reconretriagepacker_analystdotnet_decompileLESSONS_LEARNED#6 already said each analysis agent must prepare its own engine.It had been applied to the two agents someone noticed. A test now asserts the
property across all four — that is the shape of the original bug: a rule
applied case by case leaves holes invisible until a transport flakes.
The
.gitignoregap.gitignorehadenv/— the virtualenv directory pattern — but nothingmatching a file named
env. One turned up in a working tree holding liveprovider keys, a single
git add -Aaway from this now-public repository.Now covered: the bare name, the dotted forms (
.env.local) and the suffixedforms (
prod.env), with.env.examplenegated back in because it is thetemplate and must stay tracked.
env.env.env.localprod.env.env.exampleenv/x.pyVerified that no credential value appears in this repository's files, in its
history, or in this diff.
Testing
make check: 2469 passed, 1 skipped. 12 new tests, no cluster needed.Live validation still outstanding — expect zero
Optional MCP server unavailable; areplacing a stale port-forwardatreason="probe failed"isthe fix working.