Repository navigation
CI performance dashboard #1182
Description
Maintained by the CI profiler routine, which runs every 4h. Each run replaces this body and carries the history rows over.
Headline (run 2026-10-26 08:18 UTC)
Window: 2026-10-25 08:18 → 2026-10-26 08:18. 71 workflow runs (all listed, 1 page): 37 PR CI, 31 Audit GHA Workflows, 1 nightly (10-26), 2 ledger pushes. 0 merges in 24 h: the last merge was #1705 at 10-24 02:54, so the queue has been idle 53.4 h. Since 04:16: 1 full PR CI run (agent/fix-pdm-vendored-stale-install), 4 path-filtered stub runs, 4 Audit GHA Workflows runs and the 10-26 nightly. No merge_group, push-to-main or dispatch runs.
PR feedback 4.57 min; nightly red a 4th night; nothing new over the bar
The one full PR run reached ci-ok in 4.57 min (40273875744), 33.9 Linux job-min over 31 jobs, max queue wait 0.02 min. Critical path unchanged: coverage (2) 3.67, coverage (1) 3.50, Gradle redirect leg 2.88, pip VEX leg 2.57. Over 24 h, 13 green full PR runs: p50 4.58, p90 4.75 (range 4.38–4.77).
The 10-26 nightly failed again on test (windows-latest, 1) in Run tests (run 40276353917, job). That's the 4th night in a row. The failing shard ran 21.85 min against 13.28 for shard 2. The fix, #1710 (#1689), is still open and unmerged. Nightly cost: 92.5 L / 50.2 W / 30.4 M job-min. This is a correctness issue the CI janitor owns, not a CI-time finding.
Nothing new clears the 30 job-min/day or 2-min bar, so no issue was filed or commented on. The settings issues (#1670, #1248, #1512) can't be re-measured until the queue moves again.
Sample: 71 runs listed (1 page). Jobs (filter=all) for the 1 full PR run and the nightly. 0 job logs. GraphQL not used. About 10 REST requests.
| metric | 24h |
|---|---|
| PRs merged | 0 (last was #1705 at 10-24 02:54) |
| Merge queue enqueue → merge | — (no merges; last 4 took 5.13–6.85, carried) |
merge_group created → ci-ok (success) |
— (no mg runs since 10-24 02:49; 4.48 p50 with #1704, carried) |
PR run created → ci-ok |
4.57 since 04:16 (n=1). 24 h p50 4.58, p90 4.75 (n=13 green full runs, 4.38–4.77) |
push to main created → ci-ok |
— (none in window) |
| Nightly | 10-26 failed (Windows shard 1, 4th night in a row; fix #1710 still open) |
| Linux queue wait p50 / p90 | Depot 0.02 / 0.02 (n=122, max 0.03) |
| Windows queue wait p50 / p90 | 0.03 / 0.03 (nightly, n=5) |
| macOS queue wait p50 / p90 | 0.05 / 0.05 (nightly, n=4) |
| Job-min/day (Linux / Windows / macOS) | ~0.55k / 0.05k / 0.03k (13 green full PR runs ~455 L; nightly 10-26 92.5 L / 50.2 W / 30.4 M) |
Merge-queue settings (ruleset 24668460): build 5 / merge 5, min 1, min wait 5 min, ALLGREEN, timeout 90, SQUASH. Not re-read this run (no merge_group activity).
History
| run (UTC) | merges/24h | MQ enq→merge p50/p90 (min) | mg CI success p50 | PR push→ci-ok p50/p90 | mac queue p50/p90 | linux queue p50/p90 | job-min/day L/W/M |
|---|---|---|---|---|---|---|---|
| 2026-10-26 08:18 | 0 PRs merged (queue idle 53.4 h) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.57 (n=1 green full run since 04:16; 24h 4.58 / 4.75, n=13) | 0.05 / 0.05 (nightly, n=4) | 0.02 / 0.02 (Depot, n=122, max 0.03) | ~0.55k / 0.05k / 0.03k (since 04:16 PR 33.9 L in 1 full run plus 4 stub and 4 audit runs; nightly 10-26 92.5 L / 50.2 W / 30.4 M (failed, Windows shard 1); the 10-25 nightly rolled out) |
| 2026-10-26 04:16 | 0 PRs merged (queue idle 49.4 h) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.55 (n=3 green full runs since 00:16, 4.53–4.57; 24h 4.58 / 4.77, n=13) | — (no macOS jobs in slice; ≤0.03 carried) | 0.01 / 0.01 (Depot, n=93, max 0.01) | ~0.55k / 0.05k / 0.03k (since 00:16 PR 106.7 L in 3 full runs (34.95, 35.63, 36.15) plus 4 stub and 6 audit runs) |
| 2026-10-26 00:16 | 0 PRs merged (queue idle 45.4 h) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.65 (n=3 green full runs since 20:15, 4.48–4.73; 24h 4.58 / 4.77, n=11) | — (no macOS jobs in slice; ≤0.03 carried) | 0.02 / 0.02 (Depot, n=93, max 0.02) | ~0.48k / 0.05k / 0.03k (since 20:15 PR 108 L in 3 full runs (35.5, 35.8, 36.7) plus 4 stub and 7 audit runs) |
| 2026-10-25 20:15 | 0 PRs merged (queue idle 41.4 h) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.58 (n=3 green full runs since 16:15, 4.38–4.77; 24h 4.56 / 4.77, n=10) | — (no macOS jobs in slice; ≤0.03 carried) | 0.02 / 0.02 (Depot, n=93, max 0.03) | ~0.44k / 0.05k / 0.03k (since 16:15, PR 104 L in 3 full runs (34.8, 34.3, 35.0) plus 4 stub and 4 audit runs; the #1716 dispatch rolled out) |
| 2026-10-25 16:15 | 0 PRs merged (queue idle 37.4 h; 19 non-draft PRs open) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.73 (n=1 green full run since 12:15, #1723; 24h 4.55 / 4.9, n=12) | — (no macOS jobs in slice; ≤0.03 carried) | 0.02 / 0.02 (Depot, n=31, max 0.02) | ~0.65k / 0.10k / 0.07k (recounted over full window; since 12:15 PR 35.6 L in 1 full run + 4 stub + 4 audit runs) |
| 2026-10-25 12:15 | 0 PRs merged (queue idle 33.4 h; 18 non-draft PRs open) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.48 / 4.58 (n=2 green full runs since 08:15, both #1722; 24h ≈4.5 / 4.9) | — (no macOS jobs in slice; ≤0.03 carried) | 0.02 / 0.02 (Depot, n=62, max 0.02) | ~0.42k / 0.05k / 0.03k (est.; since 08:15 PR 67.6 L in 2 full runs (34.5, 33.1) + 9 stub/audit runs; the 10-24 08:15–12:15 PR slice (~70 L) rolled out) |
| 2026-10-25 08:15 | 0 PRs merged (queue idle 29.4 h; 17 non-draft PRs open) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.85 (n=1 green full run since 04:16; 24h ≈4.5 / 4.9) | ≤0.03 (nightly macOS jobs) | 0.02 / 0.02 (Depot, n=31 PR + nightly Linux legs, max 0.02; Windows max 0.05) | ~0.42k / 0.08k / 0.05k (est.; since 04:16 PR 35.7 L in 1 full run + 8 stub/audit runs, nightly 88.5 L / 51.0 W / 33.4 M (failed, Windows shard 1); the 10-24 04:16–08:16 nightly + #1710 dispatch slice (~655 L / 148 W / 119 M) rolled out) |
| 2026-10-25 04:16 | 0 PRs merged (queue idle 25.4 h; 16 non-draft PRs open) | — (none; last 4: 5.13–6.85 carried) | none since 10-24 02:49; 4.48 with #1704 (carried) | 4.48 (n=1 green full run since 00:16; 24h ≈4.5 / 4.9) | — (no macOS jobs in slice; 0.05 / 0.05 carried) | 0.02 / 0.02 (Depot, n=31, max 0.02; slow-start 0/31) | ~0.95k / 0.18k / 0.14k (est.; since 00:16 PR 35.0 L in 1 full run + 6 stub/audit runs; the 10-24 00:16–04:16 mg/push slice rolled out) |
| 2026-10-25 00:16 | 4 PRs merged (0 since 02:54; queue idle 21.4 h; the 10-23 22:15–22:56 merges rolled out) | 5.9 / 6.85 (n=4, all 02:38–02:54) | 4.48 with #1704 (n=4 incl. 6.35 pre-#1704) | 4.47 / 4.52 (n=2 green full runs: 4.42, 4.52) | — (no macOS jobs in slice; 0.05 / 0.05 carried) | 0.02 / 0.02 (Depot, n=62, max 0.02; slow-start 0/62) | ~1.55k / 0.20k / 0.16k (est.; since 20:16 PR 68.3 L in 2 full runs (34.1, 34.2) + 7 stub/audit runs; the 10-23 20:16–10-24 00:16 slice (~520 L / 12.6 W) rolled out) |
| 2026-10-24 20:16 | 9 PRs merged (0 since 02:54; queue idle 17.4 h; the 10-23 17:57/18:04 merges rolled out) | none since 04:16; last 4: 5.13–6.85 (carried) | none since 02:49; 4.48 with #1704 (carried) | 4.80 / 5.45 (n=5 green full runs: 4.48, 4.50, 4.80, 4.88, 5.45; 1 run cancelled after a Depot go env hang) |
0.05 / 0.05 (CI dispatch, n=4) | 0.02 / 0.02 (Depot, n=278, max 0.08; slow-start 5/278) | ~2.0k / 0.21k / 0.16k (est.; since 16:16 PR 215.5 L in 6 full runs (34.3, 33.9, 43.2 incl. 12-min hung coverage (1), 34.8, 35.0, 34.3); CI dispatch on #1716 98 L / 51 W / 38 M (failed, Windows shard 1); the 10-23 16:16–20:16 slice (~320 L / 5.5 W) rolled out) |
| 2026-10-24 16:16 | 11 PRs merged (0 since 02:54; queue idle 13.4 h; the 10-23 14:38 burst rolled out) | none since 04:16; last 4: 5.13–6.85 (carried) | none since 02:49; 4.48 with #1704 (carried) | 4.53 / 4.68 (n=3 green full runs: 4.37, 4.53, 4.68; 1 red PR-own hosted-e2e failure at 6.55) |
— (no macOS jobs in slice; 0.05 / 0.07 carried) | 0.02 / 0.02 (Depot, n=123, max 0.02; slow-start 1/123) | ~2.0k / 0.17k / 0.12k (est.; since 12:16 PR 138.5 L in 4 full runs (35.3, 33.2, 34.8, 35.3) + 1 single-job run; the 10-23 12:16–16:16 burst slice (~690 L / 32 W) rolled out) |
| 2026-10-24 12:16 | 24 PRs merged (0 since 08:16; queue idle; agent PRs #1708/#1710 still open) | none since 04:16; last 4: 5.13–6.85 (carried) | none since 04:16; 4.48 with #1704 (carried) | 4.54 / 4.62 (n=2 full runs since 08:16: 4.45, 4.62; 08:16 slice 4.45–4.95) | — (no macOS jobs in slice; 0.05 / 0.07 carried) | 0.02 / 0.02 (Depot, n≈100, max 0.02) | ~2.6k / 0.20k / 0.12k (est.; since 08:16 PR 69.7 L in 2 full runs (35.0, 34.7) + 14 stub/audit runs; the 10-23 08:16–12:18 slice (~130 L) rolled out) |
| 2026-10-24 08:16 | 24 PRs merged (0 since 04:16; queue idle; 4 PRs open from agents) | none since 04:16; last 4: 5.13–6.85 (carried) | none since 04:16; 4.48 with #1704 (carried) | 4.52 / 4.95 (n=3 full first-attempt runs, 4.45–4.95; all with sharded coverage) | 0.05 / 0.07 (nightly + dispatch, n=58) | 0.02 / 0.02 (Depot, n≈520, max 0.03) | ~2.7k / 0.20k / 0.12k (est.; since 04:16 PR 140 L in 4 full runs, nightly 251 / 74 / 64 (failed, Windows shard 1, core --lib), CI dispatch on #1710 264 / 74 / 55 (green)) |
| 2026-10-24 04:16 | 24 PRs merged (4 since 00:16: #1701, #1704, #1703, #1705; 0 evictions, 0 mg failures) | 5.9 / 6.85 since 00:16 (n=4, 5.13–6.85) | 4.48 with #1704 (n=3, 4.47–4.93); 6.35 for the one entry built before it | 6.70 (n=1, pre-#1704); re-run attempt with shards 4.67 | — (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 (Depot, n≈230, max 0.03) | ~2.9k / 0.21k / 0.13k (est.; since 00:16 mg 144 L / 7.6 W in 4 runs, push 16 L in 2 runs, PR 90 L; Depot slow-start 0%) |
| 2026-10-24 00:16 | 21 PRs merged (5 since 20:16: #1676, #1697, #1696, #1700, #1699; 0 evictions, 0 mg failures) | 6.98 / 8.1 since 20:16 (n=5, 6.95–8.77) | 6.47 (n=5, 6.18–8.15; 6.18–6.47 with #1697) | 6.37 / 7.03 (n=9 full runs, 4.75–7.03; #1704's coverage-shard runs 4.75 / 6.00) | — (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 (Depot, n=460, max 0.15) | ~2.9k / 0.20k / 0.13k (est.; since 20:16 mg 176 L / 12.6 W in 5 runs, push 32 L in 4 runs, PR 314 L in 9 full runs; Depot slow-start 4% (19/460), 14 of them in one PR run) |
| 2026-10-23 20:16 | 20 PRs merged (2 since 16:16: #1695, #1680; 0 evictions, 0 mg failures; the 10-22 freeze aftermath rolled out of the window) | 8.1, 10.75 since 16:16 (n=2) | 7.35 / 10.03 (n=2; 10.03 on slow hosts) | 6.8 / 9.3 (n=3 full runs, 6.57–9.90) | — (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 (Depot, n=197, max 0.08) | ~3.0k / 0.20k / 0.13k (est.; since 16:16 mg 96 L / 5.5 W in 2 runs, push 32 L in 2 runs, PR ~194 L in 5 full runs; Depot slow-start 48% (94/197), mg #1680 100%) |
| 2026-10-23 16:16 | 41 PRs merged (13 since 12:18, all 14:38–14:57 in one burst; 0 evictions, 0 mg failures) | 12.8 / 15.0 since 12:18 (n=13; first 5 entries 8.1–8.9, entries 6–12 12.2–17.4 waiting 4.3–9.4 min for a build slot) | 7.68 (n=13, p90 8.13); after #1682 6.50 / 6.73 | 8.3 / 9.1 (n=5 full runs, 6.97–9.07) | — (no macOS jobs in slice; 0.03 / 0.05 carried) | 0.02 / 0.02 (Depot, n=603, max 0.22) | ~5.6k / 0.33k / 0.13k (est.; since 12:18 mg 509 L / 32 W in 13 runs, push 92 L in 9 runs, PR ~88 L in 5 full runs; Depot slow-start 4% (26/616)) |
| 2026-10-23 12:18 | 28 PRs merged (0 since 00:56; queue empty; no nightly or dispatch in slice) | none since 00:56; 24h 133 / 175 carried | none since 04:16 (last 7.9) | 7.7 / 8.1 (n=4 full runs since 08:16, 7.30–8.12) | 0.03 / 0.05 (carried) | 0.02 / 0.03 (Depot, n=116, max 0.05) | ~5.1k / 0.30k / 0.13k (est.; since 08:16 PR ~130 L in 4 full runs + 7 stubs; Depot slow-start 0% (0/116)) |
Open ci-perf issues by ROI
ROI on one scale: (thousands of job-min/day weighted Linux ×1, Windows ×2, macOS ×3, plus 1 per critical-path or PR-latency minute) × confidence ÷ effort (S=1, M=2, L=3).
| ROI | issue | saving | status |
|---|---|---|---|
| ~14 | #1670 (settings) 90-min check timeout let a stuck check freeze main for 77 min | ~46 min sooner recovery per stuck-check incident, for every waiting entry (7 on 10-22) | No recurrence since 10-22 20:16 (all mg runs green; none since 10-24 02:49). Timeout still 90. Triaged p3 and needs a repo admin. ROI: 46 MQ-min × 0.3 ÷ 1. |
| ~11 | #1248 (settings) merge-queue build concurrency | 4–9 min per entry past the 5th in a burst; 44 entry-min in the 10-23 14:30 burst | No burst since 10-23 16:16. With mg ci-ok ~4.5 min, each slot turns over ~2 min faster, which shrinks this saving. Needs a human decision. |
| ~2.5 | #1512 (settings) slow Depot hosts | ~2–4 min per merge on a slow host, plus lost-runner evictions | Slow-start 0/62 since 20:16 (5/278 in the 16:16 slice, plus 1 hung go env step (11.5 min) in coverage (1), not recurred). Intermittent. Last numbers posted 10-24 00:18; nothing new worth a comment. Needs a human to raise it with Depot. |
| ~1 | #1720 (coverage tradeoff) Windows-only test regressions reach main because Windows tests run only nightly | avoids ~0.2–0.3k job-min of fix churn per regression (5 in 3 nights), at a cost of +3–5 critical-path min per entry (option 1) | Filed 10-25 03:43 by the CI janitor, agent:needs-human. Costed by the profiler on 10-25 04:16 (comment). The path-filtered option 1 is cheapest. Needs a maintainer decision. |
Closed since last run: none. Filed: none. Commented: none. There are no fresh numbers: no merge_group activity since 10-24 02:49 (53.4 h), so #1670 / #1248 / #1512 are unchanged. #1720 was costed 10-25 04:16 and is still waiting on a maintainer.
Open ci-perf / ci-janitor PRs: none labelled ci-perf. ci-janitor: #1708 (draft runs cancelling real PR CI) and #1716 (Pipenv pylock test follows uv), both still open and non-draft (not merged; queue idle since 10-24 02:54, 53.4 h). Nightly fix #1710 (#1689) is still open.
Merged ci-perf / ci-janitor PRs (last 7 days) and measured effect
| PR | merged | before → after |
|---|---|---|
| #1704 shard coverage across two runners (ci-perf, self-picked from dashboard) | 10-24 02:38 | Delivered, beat its estimate. merge_group ci-ok with the shards: 4.47, 4.48, 4.93 (n=3; run, run, run) vs 6.18–6.47 after #1697 and before #1704: about −1.8 min per merge (expected −1.5). Shards take 3.78–4.62 min (was one 5.88–6.17 job). Cost +0.7–1.4 Linux job-min per mg run (35.9–36.6 vs 35.2), less than the +4 expected. Push to main ci-ok 3.03 / 3.08 (was 3.12–3.27). PR push→ci-ok since 04:16: 4.45–4.95 (n=3) vs 6.37 p50 before. |
| #1697 Gradle agent suites in their own e2e leg (self-picked from dashboard) | 10-23 22:18 | Delivered. The Gradle legs now take 2.82–2.98 and 1.77–1.97 min (was 4.58–4.92 combined, 7.40 on a slow host). ci-ok in the 3 merge_group runs that include it: 6.18 / 6.23 / 6.47 vs 6.50–7.58 on fast hosts before (about −0.3 to −1 min). Gradle is off the critical path; coverage now sets it. |
| #1682 start coverage and e2e-build without waiting on clippy (#1681) | 10-23 14:54 | Delivered. In merge_group runs, coverage now starts at 0.20–0.30 min (was 1.43–1.47). ci-ok with #1682: 6.50, 6.73, 7.35, 7.58, 10.03 (n=5; p50 7.35, the 10.03 on 100%-slow hosts) vs 7.53–8.15, p50 7.76 before. On fast hosts ~1.1 min saved against 1.3 expected. |
| #1686 Windows nightly path asserts (ci-janitor) | 10-23 14:38 | Partly delivered. In the 10-24 nightly Windows shard 2 passed, but shard 1 still failed on -p socket-patch-core --lib (job). Follow-up #1710 (#1689) is open and green in dispatch. |
| #1690 gem old-Bundler legs (follow-up to #1687) | 10-23 14:54 | Delivered. No gem leg failures in the 10-24 nightly (its only failure was Windows shard 1). |
| #1692 compatibility workflows off Node 20 pins (ci-janitor) | 10-23 14:47 | Maintenance; no CI-time effect expected. |
| #1673 retry the yarn-classic corepack probe (#1671) | 10-23 00:13 | Before: 2 of 40 tests failed on one DNS timeout and evicted #1653 (~94 job-min). After the merge: 0 yarn-classic failures in 28 jobs (7 full runs through 04:16, then 21/21 green since). Holding. |
| #1663 fix uv settings test on Windows nightly (ci-janitor) | 10-22 19:29 | Did not make the Windows nightly green. The 10-23 nightly failed both Windows shards (cli --test user_patches, core --lib, path-form asserts). Follow-up #1686 passed both shards in its dispatch run. |
| #1652 keep transport cause in registry errors | 10-22 17:38 | Diagnostics; no CI-time effect expected. |
| #1654 retry crates.io in coverage's cargo e2e | 10-22 17:12 | 0 infra coverage failures in 53 merge-group + 19 push runs + 10 PR runs since 04:16. The one 04:16–08:16 failure was a PR's own new test. The 10-22 21:25 PR failure is carried. Holding. |
| #1643 fix update-lock flake | 10-22 17:09 | 0 recurrences since. Too early to call. |
| #1647 pin floating toolchains (#1645) | 10-22 17:03 | No toolchain-drift red main since (7 h). Needs a week to judge. |
| #1659 runner image's PHP in composer e2e | 10-22 17:03 | 0 Setup PHP failures since (was 1 re-run on 10-21). Composer leg p50 unchanged. |
#1666 skip cargo shapes on failed --precise pin |
10-22 17:01 | Robustness; no CI-time effect expected. |
#1620 ci-watch.sh works without GraphQL (ci-janitor) |
10-20 19:02 | Tooling; no CI-time effect expected. |
| #1634 keep reuse self-test notices out of CI logs (ci-janitor) | 10-20 19:02 | Log hygiene; no CI-time effect expected. |
| #1590 move action pins off deprecated Node 20 (#1588, ci-janitor) | 10-17 14:47 | Maintenance. No measurable change, as expected. |
Method notes
- GraphQL and
search/issuesare blocked in this environment. Merge-queue state comes from REST timelines of merged PRs. "First enqueue" means the PR's firstadded_to_merge_queueevent in the window. "Slot wait" = lastadded_to_merge_queue→ that entry's merge_groupCIruncreated_at. - Jobs are fetched with
filter=all. Queue wait = jobstarted_at−created_at; skipped jobs are excluded. "Start-up gap" = jobstarted_at→ first stepstarted_at; over 15 s counts as a slow host. - Critical path = merge_group run
created_at→ci-okcompleted_atof the latest attempt. - "Discarded green runs" = successful merge-group runs whose entries were rebuilt on a new base before merging.
- This run (10-26 08:18): all 71 runs in the window were listed. Job-minutes come from run durations plus job detail for the 1 new full PR run and the nightly.
- Previous run (10-26 04:16): all 71 runs in the window were listed.
Generated by Claude Code
Activity
mikolalysenko commentedon Oct 8, 2026
[agent] There are now two open dashboard issues: this one and #1175, both titled "CI performance dashboard" and labeled
ci-perf-dashboard. The body says it is maintained in place, so the profiler routine should update one issue rather than open a new one each run. I'm not closing either one, because I can't tell which issue the routine will update next. Whoever owns the profiler should close the copy it no longer updates.Generated by Claude Code