Repository navigation
fix(signals): Ctrl-C and SIGTERM do not stop the loop - #345
Conversation
The SIGINT/SIGTERM trap called cleanup(), which returns instead of exiting, so bash resumed the main loop: the operator saw "Ralph loop interrupted. Cleaning up..." and the loop kept running. Five faults in one control: 1. trap cleanup SIGINT SIGTERM — the handler never terminated. 2. The comment claimed "EXIT trap handles natural termination". There is no EXIT trap anywhere in the file; that trap line was the only one. 3. The _CLEANUP_DONE reentrancy guard, documented as covering an "EXIT + signal combination" that does not exist, made every signal after the first a silent no-op. 4. cleanup() read $? as trap_exit_code. In a trap handler that is the last completed command's status, not 128+n, so the "interrupted" status was recorded only when the preceding command happened to fail. 5. The claude child is backgrounded with a `local claude_pid`, invisible to the handler, so a SIGTERM to the script left the child running. Fix: a dedicated on_signal() that restores default dispositions first (so a second Ctrl-C always kills even if cleanup hangs), stops the child via a new CLAUDE_CHILD_PID global, runs cleanup, then re-raises the signal so the process dies with the correct 128+n status. _INTERRUPTED makes the interrupted-status record deterministic. Proof: 7/7 in --dry-run (no API call), each patched case paired with the unpatched original as a known-false control. SIGINT to pgroup patched DEAD 1s (130) · original ALIVE at 15s SIGTERM to pid patched DEAD 2s (143) · original ALIVE at 15s SIGINT x4 patched DEAD on first · original ALIVE, Loop frankbria#3 no signal patched still looping — normal path unaffected The original's control log reproduces the reported symptom exactly: "interrupted. Cleaning up..." followed by "Completed Loop frankbria#1". Not covered: dry-run returns before spawning a child, so the CLAUDE_CHILD_PID kill leg is reasoned, not measured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five tests, each verified to FAIL against the unpatched loop and pass
against the fix:
- SIGTERM terminates the loop (exit 143)
- SIGINT terminates the loop (exit 130)
- repeated signals do not disarm the handler
- the bare `trap cleanup SIGINT SIGTERM` form is gone
- the child pid is reachable from the handler
All run under --dry-run, so no API calls are made.
Two harness notes worth keeping, both of which produced false results
before they were understood:
- A background job started from a non-interactive shell inherits SIGINT
as SIG_IGN, and bash cannot re-trap an inherited-ignored signal. A
naive harness shows the loop surviving Ctrl-C whether or not the bug
is present, which proves nothing. The perl wrapper resets SIGINT to
default before exec, as an interactive terminal would.
- `wait` is used rather than a kill -0 poll: a dead-but-unreaped child
still answers kill -0. A watchdog bounds the wait so a hung loop
cannot wedge the suite.
A sixth test was written and dropped: it asserted that no further
iteration is logged after the "interrupted" line, but that line is only
printed when the preceding command happened to fail, so under --dry-run
it never appears and the assertion matched nothing. It passed against
the buggy code and was therefore worthless.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ctrl-C returned the prompt but the agent kept working and kept billing.
Measured mid-work under a real pty:
loop pid 94095 pgid 94095 <- foreground group; gets the Ctrl-C
bash pid 94100 pgid 94095 <- $! captured THIS, not the agent
gtimeout pid 94107 pgid 94107 <- gtimeout setpgid()s ITSELF
claude pid 94112 pgid 94107 <- the agent, in that other group
portable_timeout is a shell function, so backgrounding it forks a subshell;
bash sometimes execs gtimeout in its place and sometimes does not, varying
with execution context, so $! does not reliably name the agent. Ctrl-C
reaches only the foreground group, so the agent never sees it; on_signal
then TERMs the subshell and gtimeout plus the agent are orphaned and run on.
A process-group kill is not the answer either: the agent is not in our
group, so -$PGID signals the loop itself and spares the agent.
Add list_descendants() and kill_tree(), and call kill_tree from on_signal.
Membership is proven by parentage and enumerated BEFORE anything is
signalled -- once the root dies its descendants reparent to launchd and no
later walk can find them. Deepest-first TERM, 5s grace, then SIGKILL to
survivors; never signals itself, an ancestor, or a process group.
Written for bash 3.2, which #!/bin/bash on macOS means it must be: no
mapfile, no readarray, no array expansions that abort under set -u.
Evidence, control vs fixed:
real agent, real Ctrl-C 3 files after the stop, ran to completion,
$0.242 -> 0 files, killed at 5 turns, $0.105
zero-cost stub, repeated fails 3/3 -> passes 3/3
library selftest (3.2) control fails as it must -> 6/6
Not addressed here: CLAUDE_CHILD_PID is assigned only in the background
branch, so the child-kill leg is a no-op under --live, where the agent runs
as a foreground pipeline and is never registered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven tests, each verified to FAIL against unpatched main at 0c1d7bf and pass against the fix: list_descendants finds a child and a grandchild through a wrapper a single-pid TERM leaves the tree alive (known-false control) kill_tree clears the whole tree from a live root kill_tree reaches a descendant in a DIFFERENT process group kill_tree does not kill its own caller kill_tree escalates to SIGKILL past a TERM-ignorer the signal handler stops the tree, not a single pid They source the shipped ralph_loop.sh in a subshell, so the functions under test are the ones that ship and the loop's own traps never land in the bats shell. Trees are built from sleep; no API calls, no cost. The control is inside the suite rather than in a commit message: test 2 asserts the OLD single-pid TERM still fails to stop a tree. If it ever goes green, the kill_tree tests are proving nothing and it fails loudly. Test 4 is the production shape. gtimeout setpgid()s itself, so the agent is not in the loop's group; the test asserts the fixture actually straddles a group boundary before asserting the tree died, or it would pass on a fixture that never tested the thing. Two harness notes, both of which produced wrong results first: - Test 5 originally passed against unpatched main. kill_tree does not exist there, so nothing died and the caller trivially survived. It now also asserts the target died. A test that goes green against the buggy code is worthless -- the same fault this suite dropped a sixth test for. - Cleanup must not be a test's last command: killing an already-dead pid exits non-zero and fails the test for the wrong reason. Suite: 1286 tests, 13 failures, the same 13 pre-existing platform failures present on unpatched main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
frankbria
left a comment
There was a problem hiding this comment.
Reviewed and verified — the root cause is confirmed on main (signal trap calls cleanup(), which returns, so the loop resumes; it also tears down the docker/e2b sandbox before resuming). This is the highest-priority fix in the open queue. Merged onto current main locally: full unit + integration suite passes (1291/1291), no conflicts.
One non-blocking note for a follow-up: in --live/--monitor mode Claude runs as a foreground pipeline rather than a backgrounded child, so CLAUDE_CHILD_PID isn't set, and bash defers a trapped SIGTERM until the foreground command finishes. Ctrl-C still works (the terminal signals the whole foreground group), but an external kill -TERM in live mode may not take effect until the current Claude call ends or times out.
|
Thanks for this, and sorry it sat so long. We're taking this PR as the basis for the signal-handling fix and closing the parallel #364. Yours is the more complete version: the portable Its CI run expired unapproved, so I'm closing and reopening this PR to start a fresh run. One known gap (you already call it out in the description): in |
|
Rebased locally onto current main with no conflicts. Full suite passed (1384 unit+integration, 28/28 e2e). Merging with a rebase merge so your four commits keep their authorship. Thanks, Sofoklis-byte. The live/monitor-mode follow-up is tracked in #394. |
Summary
Ctrl-C does not stop Ralph. It prints
Ralph loop interrupted. Cleaning up..., then the loop continues to the next iteration. Repeated Ctrl-C does nothing at all, andSIGTERMdoes not stop it either. In the run that surfaced this, onlykill -9ended the process — about seven minutes after the first stop attempt.The reassuring log line is the dangerous part: an operator watching an autonomous loop believes it has stopped when it has not.
Root cause
trap cleanup SIGINT SIGTERMinstalls a handler that returns instead of exiting.cleanup()ends with:# No exit here — EXIT trap handles natural terminationThere is no EXIT trap. That
trapline is the only one in the file. The comment describes a handler that does not exist, so nothing ever terminates the process, and bash resumes the loop as soon as the handler returns.Four further faults sit behind that one:
cleanup()returns; the signal handler therefore never terminates.EXIT trapthe comment relies on does not exist anywhere in the file._CLEANUP_DONEreentrancy guard — documented as covering an "EXIT + signal combination" that cannot occur — makes the second and every later signal a silent no-op. This is why repeated Ctrl-C cannot recover.cleanup()reads$?astrap_exit_code. In a trap handler that is the last completed command's status, not128+n, so theinterruptedstatus is recorded only when the preceding command happened to fail. Under--dry-runit never prints.claudeis backgrounded and its pid is alocal claude_pid, invisible to the handler, so aSIGTERMsent to the script leaves the child running.Fix
A dedicated
on_signal()replaces the bare trap:CLAUDE_CHILD_PID— this leg proved insufficient and is corrected by28cf3ca; see Follow-up below;cleanup()for teardown, unchanged in meaning;128+nstatus rather than falling out of a loop — anything supervising Ralph now sees a real interrupt;_INTERRUPTED, which makes the interrupted-status record deterministic rather than dependent on the previous command's exit code.cleanup()keeps its single-run guard and its existing behaviour. Normal (unsignalled) operation is unchanged.Tests
tests/integration/test_signal_handling.bats— five tests, each verified to fail against the unpatched loop and pass against the fix. All run under--dry-run, so they make no API calls.Two harness notes are recorded in the test file, because both produced false results before they were understood:
SIG_IGN, and bash cannot re-trap an inherited-ignored signal. A naive harness shows the loop surviving Ctrl-C whether or not the bug is present — it proves nothing. The tests reset SIGINT to default beforeexec, as an interactive terminal does.waitis used rather than akill -0poll, because a dead-but-unreaped child still answerskill -0. A watchdog bounds the wait so a hung loop cannot wedge the suite.A sixth test was written and dropped: it asserted that no further iteration is logged after the
interruptedline, but per fault 4 that line is not printed under--dry-run, so the assertion matched nothing and passed against the buggy code. It is mentioned here so it is not re-added later in good faith.Follow-up — the child-kill leg was insufficient (
28cf3ca)The original fix reached the child pid, and the tests above assert exactly that: the child pid is reachable from the handler. Reaching it turned out not to be enough. In production the operator pressed Ctrl-C, got the prompt back, and the agent wrote a file 2s later, committed 17s later, and finished 47s after the stop, billing the call in full.
Measured mid-work under a real pty:
portable_timeoutis a shell function, so backgrounding it forks a subshell; bash sometimes replaces that subshell with an exec ofgtimeoutand sometimes does not, varying with execution context.$!therefore does not reliably name the agent. Two consequences:TERMing$!kills the subshell;gtimeoutand the agent are orphaned and keep working.A process-group kill is not the answer either: the agent is not in our group, so
-$PGIDsignals the loop itself and spares the agent. (Measured — a candidate doing this killed the supervisor and left the target running.)28cf3caaddslist_descendants()andkill_tree()and calls the latter fromon_signal(). Membership is proven by parentage and enumerated before anything is signalled — once the root dies its descendants reparent tolaunchdand no later walk can find them. Deepest-firstTERM, 5s grace, thenSIGKILLto survivors; it never signals itself, an ancestor, or a process group. Written for bash 3.2, which#!/bin/bashon macOS requires: nomapfile, noreadarray, no array expansions that abort underset -u.Control vs fixed:
28cf3ca)Regression tests (
99f2323).tests/integration/test_process_tree_kill.bats— seven tests, each verified to fail against unpatchedmainat0c1d7bfand pass against the fix. They source the shippedralph_loop.shin a subshell, so the functions under test are the ones that ship; trees are built fromsleep, so there are no API calls.The control lives inside the suite: test 2 asserts the OLD single-pid TERM still fails to stop a tree, so if it ever goes green the
kill_treetests are proving nothing and it says so. Test 4 asserts the fixture actually straddles a process-group boundary before asserting the tree died.main—kill_treedoes not exist there, so nothing died and the caller trivially survived. It now also asserts the target died. This is the same fault the sixth signal test was dropped for, and it is recorded rather than quietly fixed.What this evidence is still not. The behavioural proof of the original defect came from an external pty harness and a faithful miniature of the loop's spawn-and-signal architecture. The control has never been exercised against a live
ralph_loop.shend to end.Known gap, not addressed here.
CLAUDE_CHILD_PIDis assigned only in the background branch. Under--livethe agent runs as a foreground pipeline and is never registered, so the child-kill leg is a no-op in that mode.Test suite status
Re-measured on macOS at
28cf3ca, not carried from an earlier run:npm testat99f2323npm testat unpatchedmain0c1d7bfnpm run test:e2eat28cf3caThe 13 failures are identical in name and number across both runs, so this PR introduces none of them. They are platform failures (
head: illegal line count -- -1, a Linux-onlynotify-sendcase, a BSDstatfallback case), none in code this PR touches. The count differs by 12 because this PR adds 5 signal tests and 7 process-tree tests.The e2e suite runs
ralph_loop.shas a real subprocess and includes termination signal during execution records interrupted status and preserves call count, which exercises the signal path this PR changes.Reproducing the bug