fix(branch): converge merges promptly via drain-ETA cadence and never-started-count rekick

[thedumbtechguy]

What & why

BranchMergeJob (the poller that drives a fan-out's merge to completion) had two defects, surfaced by the fan-out scale test:

1. Cadence overshoot — the old count-based cadence polled slowest (max_interval, ~5 min) exactly when a fast-draining backlog was about to finish, so a parent could hang idle for up to 5 min after its last child completed.
2. Premature / repeated rekick — healthy but deeply-queued children were re-enqueued as if dropped, because the rekick signal keyed off total pending, which a wait_until child resuming would drop without any never-started child actually moving.

The poller holds no lock and never replays the (heavy) parent; each pass is fenced by a per-pass poll_token so a superseded chain stops quietly.

The two fixes

1. ETA-driven cadence (reschedule_delay). Poll at (pending / rate) * ETA_FRACTION, clamped [min_interval, max_interval]. Because each pass re-estimates against the shrinking remainder, cadence converges geometrically: a 500k fan-out polls flat max_interval through the long middle, then tightens to min_interval for the last few thousand — the parent is woken within ~min_interval of the last child, not up to a max_interval late. When nothing completes in an interval the fallback is motion-aware: :running holds the responsive floor (a slow/low-fan-out child is never woken late), :never_started backs off exponentially (a queued-but-unpicked straggler), :none decays to max_interval. Motion probes are computed lazily, only off the drain path.

2. Never-started-count rekick gate. Rekick is gated on the never-started count (idle & started_at nil) delta, not total pending: if that count fell since the last poll, workers are consuming the queue, so deep-queued-but-healthy children are left alone. Only a branch whose never-started count has gone flat has its stale children rekicked, and a touch per rekick debounces it to at most once per REKICK_AFTER. Blocked (failed/stalled) and waiting (idle with started_at set) children are never touched.

Supporting work

• Poller queue config — ChronoForge.config.branch_merge_queue (default :default). merge_branches enqueues the poller after dispatching children, so it must not share a large fan-out's queue or it starves behind the backlog (polls once at pending≈0, no ETA, dashboard gauge never renders). Documented as a trap in the scale-test doc.
• Persisted poll state — ActiveJob can't be queried for a scheduled poller, so each pass stamps its observable state onto the branch-log metadata (last/next_poll_at, interval, pending, never_started, spawned, rate, eta_seconds, polls, rekick counts) under a row lock + token recheck. Purely observational; replay/correctness never read it.
• Dashboard — live throughput/ETA on in-flight merges (aggregated across a multi-branch merge: summed rate, combined ETA; sub-1/s rates kept visible via round(3)); a branch-detail stats header; universal auto-refresh (the whole <main> is a data-poll-region, so every page — including detail — refreshes in place, preserving filter text/focus/scroll).
• Exact counts, no new queries — pending and never_started render exact straight from the poll metadata the poller already computes; the total spawned is immutable once a branch seals, so it's counted once and cached on the metadata (sticky), then shown exact with zero per-render cost. All fall back to the capped live count until the branch is sealed+polled. The mutable per-state chips stay capped (5000+).
• Naming — the poller-side never-started count was renamed dispatched → never_started to kill a collision with the dashboard's total column, which was in turn renamed "Dispatched" → "Spawned" (it maps to spawn_each → spawned_workflows).

Validation

• Engine 264 tests, dashboard 108 tests — all green.
• Live-driven at 20k / 100k / 500k on Solid Queue + Postgres; 500k merged flawlessly (every child completed, parent converged, constant memory). Screenshots refreshed from a live 100k drive.

🤖 Generated with Claude Code