fix(o11y): sample floors for the 5xx and LiteLLM error-rate alerts

[demtario]

Context

A deliberate production probe on 2026-09-30 showed api-5xx-rate can page on a single 500 at low traffic, because it has no minimum sample. This continues #390 (which added sample floors to preview-ready-rate and session-start-p95) and the alert-accuracy follow-up DEV-3143, without claiming to close it. I audited every rule in runner/workers/o11y/src/alerts/rules.ts and added a floor only to the two ratio rules that could fire on one event: api-5xx-rate (new FIVE_XX_MIN_REQUESTS = 100 on the adjusted total, 15 min window) and litellm-error-rate (new LITELLM_MIN_CALLS = 20 non-denied calls, 1 h window). The floors follow one principle: the smallest total at which a single error cannot exceed the threshold (1 error / 100 = 1 %, not above 1 %; 1 / 20 = 5 %, not above 5 %). Below a floor the rule is simply not firing, so an active alert resolves through the existing notify path, and the detail text says the sample is below the floor. At roughly 60 requests per minute (about 900 per 15 min) the 5xx floor will rarely bind in busy hours. No SQL changed. ADR-0041 section F.3 is updated; no other doc lists these thresholds.

Audit of every rule:

Rule	Kind	Threshold	Window	Floor before	Floor after
at-capacity-rate	count	more than 5	1 h	n/a, a count	unchanged (a count, not a ratio)
api-5xx-rate	ratio	more than 1 %	15 min	none	100 requests on the adjusted total (new)
preview-ready-rate	ratio per tier	below 97 % / 95 %	1 h	10 non-abandoned per tier (#390)	unchanged
session-start-p95	quantile	above 20 s	1 h	20 ready starts (#390)	unchanged
embed-error-rate	ratio per demo	above 20 %	24 h	more than 50 views (one error needs under 5 views to exceed 20 %)	unchanged
compile-error-doubling	day-over-day	2x	24 h	5 today-count	unchanged
snapshot-build-failed-rate	ratio per framework	above 50 %	30 min	10 failed across 3+ demos	unchanged
litellm-error-rate	ratio	more than 5 %	1 h	none	20 non-denied calls (new)
backlog-age	state	older than 2 h	n/a	n/a	unchanged (state-based)
rejected-inbox-key	state	any recent rejection	1 h	n/a	unchanged (state-based)
admission-overflow	state	any dropped	20 min	n/a	unchanged (state-based)
new-fingerprint	detection	any new	n/a	n/a	unchanged (dashboard only, no Slack)
o11y-spend-cap	state	spend at or above cap	n/a	n/a	unchanged (state-based)
alert-eval-error	synthetic	any rule failed to evaluate	n/a	n/a	unchanged (state-based)

Types of changes

• New example
• Update to an existing example
• README / documentation change
• Demo runner (runner/) change
• CI / tooling change

How was this verified?

Tests in runner/pipeline/o11y-alerts.test.mjs cover, for each changed rule, below the floor (not firing even at 100 % bad), one error exactly at the floor (not firing), at the floor above the threshold (fires), above the floor healthy (not firing), and the resolve path when a firing alert drops below the floor; the 5xx floor is also shown to use the adjusted total and the LiteLLM floor to ignore denied calls. Existing tests already seeded at least the floor, so none needed re-seeding. Revert proof: with rules.ts checked out from the base commit (23ebd18) the 6 new tests fail (3 per rule) and the other 81 pass; neutralising only the two floor guards fails the same 6. Runs: node --experimental-strip-types --test pipeline/*.test.mjs 2581 pass, 0 fail; tsc --noEmit clean in workers/o11y and workers/api; pnpm --filter @handsontable/demo-runtime build ok; wrangler deploy --dry-run --containers-rollout=none in workers/o11y ok (plain dry-run needs Docker, which is not running locally). The fake AE engine only proves the SQL passes the guard, not that Analytics Engine accepts it. Nothing here was validated in production or against a live Analytics Engine.

Checklist

• New/renamed example: not applicable, no example changed
• New example: not applicable
• Ran pnpm build (and pnpm dev) in the affected example/server-example locally: not applicable, no example changed; runner checks are listed above

Related issue(s):

1. Continues fix(o11y): sample floors for preview-ready and session-start alerts, new fingerprints to Grafana instead of Slack #390 and the alert-accuracy follow-up DEV-3143 (not closed by this PR).

---

Note

Low Risk
Alert logic only tightens false-positive paging on low traffic; thresholds and windows are unchanged once sample floors are met, with tests covering resolve behavior.

Overview
Adds minimum sample floors to the two ratio-based o11y alerts that could page on a single error at low traffic, following the same pattern as preview-ready and session-start-p95 (#390).

api-5xx-rate now requires at least 100 adjusted api.request samples in the 15-minute window (after deliberate 5xx exclusions) before the >1% threshold is evaluated; one isolated 500 no longer fires the rule. litellm-error-rate requires at least 20 non-denied gateway calls in the hour before the >5% threshold applies. Below each floor the rule is treated as not firing (so an active alert resolves normally), and rule detail text explains the shortfall.

ADR-0041 §F.3 documents the new floors. pipeline/o11y-alerts.test.mjs adds coverage for boundary behavior, adjusted totals / denied exclusion, and fire→resolve when volume drops under the floor. No Analytics Engine SQL changes.

^{Reviewed by Cursor Bugbot for commit 0afd0b3. Bugbot is set up for automated code reviews on this repo. Configure here.}