Surfaced during PR #232's CI run. Pre-existing, unrelated to that change, and it will recur.
drip.test.ts's "per-run cap" test failed once in CI on a 5000ms timeout, then passed on a re-run of the same job. Locally it completes in 267ms (27/27), and main was green six minutes earlier.
Why it's marginal by construction: the test seeds 60 users sequentially then runs 2 sweeps, all inside a 5000ms budget. Fine on a developer machine at ~267ms, but CI runners are slower, contended and variable, so the same work lands anywhere in a wide band. The test isn't wrong; its margin is thin and its cost scales with a fixture size chosen for realism.
Why it deserves fixing rather than re-running.
A timeout flake is worse than an assertion flake, because it is indistinguishable from a real performance regression.
This exact test guards a path that is about to get new predicates: the account-delete work adds deleted_at IS NULL to all three drip eligibleFor queries (PR #232). If one of those ever makes the sweep genuinely slower, it will present as this same timeout — and the established response will be "known flake, re-run", which is precisely how a real regression gets waved through.
That is not hypothetical. It already happened once tonight. The builder did the right thing: ran it locally, checked main's recent history, re-ran only the failed job, and explained the reasoning instead of silently reporting green. But that judgement should not be required every time.
Options
- Raise the timeout for this test specifically, with a comment explaining why it exceeds the default. Cheapest, keeps coverage — but widens the window where a genuine slowdown hides.
- Shrink the fixture. Cap behaviour almost certainly does not need 60 users to demonstrate; a cap of N with N+2 seeds tests the same invariant far faster.
- Seed in a batch rather than sequentially, if the sequencing is not itself under test.
The second is likely best: it removes the flake and tightens the signal, rather than trading one for the other.
Related: parachute-app #120 is the sibling flake (Welcome.test.tsx:190, a navigation race). Same family — a gate that fails at random trains everyone to merge through it, which is how a staging gate in this repo sat red for nine days while silently not running seven assertions.
Surfaced during PR #232's CI run. Pre-existing, unrelated to that change, and it will recur.
drip.test.ts's "per-run cap" test failed once in CI on a 5000ms timeout, then passed on a re-run of the same job. Locally it completes in 267ms (27/27), andmainwas green six minutes earlier.Why it's marginal by construction: the test seeds 60 users sequentially then runs 2 sweeps, all inside a 5000ms budget. Fine on a developer machine at ~267ms, but CI runners are slower, contended and variable, so the same work lands anywhere in a wide band. The test isn't wrong; its margin is thin and its cost scales with a fixture size chosen for realism.
Why it deserves fixing rather than re-running.
A timeout flake is worse than an assertion flake, because it is indistinguishable from a real performance regression.
This exact test guards a path that is about to get new predicates: the account-delete work adds
deleted_at IS NULLto all three dripeligibleForqueries (PR #232). If one of those ever makes the sweep genuinely slower, it will present as this same timeout — and the established response will be "known flake, re-run", which is precisely how a real regression gets waved through.That is not hypothetical. It already happened once tonight. The builder did the right thing: ran it locally, checked main's recent history, re-ran only the failed job, and explained the reasoning instead of silently reporting green. But that judgement should not be required every time.
Options
The second is likely best: it removes the flake and tightens the signal, rather than trading one for the other.
Related:
parachute-app#120 is the sibling flake (Welcome.test.tsx:190, a navigation race). Same family — a gate that fails at random trains everyone to merge through it, which is how a staging gate in this repo sat red for nine days while silently not running seven assertions.