Stop publishing runs whose responses were mostly errors

[MDA2AV]

rps counts a 500 exactly like a 200, and nothing between the load generator and the board ever looked at the status counts. They're parsed by every tool adapter, written into results/, carried through rebuild_site_data.py, and listed in gen_new_leaderboard_data.py's BASE_FIELDS — then read by nobody. So a framework that collapsed under load was ranked on the rate at which it produced failures.

The 21 rows that weren't measurements

framework	profile	published	errors
php-fpm	baseline-h2-1024	294,864 rps	99.2% 5xx
symfony-spawn-franken	baseline-h2-256	28,243 rps	72.9% 5xx
symfony-spawn-franken	static-h2-1024	52,582 rps	56.5% 5xx
symfony-spawn-franken	static-h2-256	63,518 rps	47.3% 5xx
symfony-spawn-franken	baseline-h2-1024	97,817 rps	36.5% 5xx
phoenix-bandit	upload-{32,256}	~700 rps	33.3% 4xx
rage	api-16-1024	9,825 rps	15.2%
rage	api-4-256	13,999 rps	8.8%
apache,caddy,pingora	pipelined-{512,4096}	0 rps	100% 4xx
traefik	pipelined-{512,4096}	0 rps	100% 5xx
fasthttp	crud-4096	0 rps	0 responses
effinitive	static-h2-1024	0 rps	0 responses
trillium-tuned	static-h2-{256,1024}	0 rps	0 responses

php-fpm is the clearest case: 294k rps of 500s placed it mid-field on baseline-h2, ahead of frameworks answering every request correctly.

The four zero rows are a different lie, and the logs say why — fasthttp crashed (a V backtrace in site/static/logs/crud/4096/fasthttp.log, plus 14,506,257 reconnects as every connect was refused); effinitive and trillium-tuned bound their listeners and then returned nothing while burning 1486–6187% CPU and 4.3–5.1 GiB. Publishing rps: 0 states "we measured zero throughput", which ranks them last on a test that never ran. Absent is the honest value, and the composite already reads absent as 0.

Changes

• rebuild_site_data.py grows failure_reason() and applies it in merge_results() to the merged list, not just the incoming batch — so a bad row already in site/data is dropped on the next rebuild even when its framework isn't re-run. This is load-bearing: site/data is generated, so hand-deleting these rows would have lasted exactly until the next --save restored them from the raw files still on the box. Threshold is 5% non-2xx/3xx, overridable via HTTPARENA_MAX_ERROR_PCT.
• benchmark.sh warns at save time on the same rule. The row and container log are still written — they're the post-mortem evidence — but a clean-looking --save shouldn't imply a result that will never reach the board.
• The 21 rows are purged, applied with the generator's own function so the purge can't drift from the rule.
• Docs: a short "Failed runs are not scored" section in composite-score.md, so a contributor whose row goes missing can find out why.

Ranking impact

Small — these rows were mostly earning little: rage −24.4 (api-4 is 20.6 of it), phoenix-bandit −9.4, symfony-spawn-franken −5.6, php-fpm −1.0, against a ~2600 ceiling. The zero rows score 0 either way. The eight pipelined proxy rows change nothing: pipelined is reference-only, and #1017 removes those frameworks outright.

Deliberately left alone

The 1–5% band stays published — roadrunner, sark, vanilla-epoll, swerver and others sit there, and at 16k connections a handful of failures is plausible rather than diagnostic. 1969 of 2017 rows had exactly zero errors, so the tail is thin either way.

Verification

Rebuilt a synthetic results tree through merge_results(): 9.1% dropped, 4.8% kept, and a pre-existing failed row purged without being re-run. The shell warning was checked against the same thresholds including the env override. 1996 published rows remain, 0 failing the rule.

🤖 Generated with Claude Code