Store results one file per framework#1034
Merged
Merged
Conversation
Closes #751. Results lived in 52 shared arrays — site/data/<profile>-<conns>.json, each holding every framework — so a saved run rewrote files that every other framework also occupies. They now live one file per framework: site/data/baseline-512.json [ {actix…}, {hyper…}, … ] -> site/data/results/actix.json { framework, results: { "baseline-512": {…} } } 108 files, 1941 rows, same rows verbatim. The conflict is real but narrower than "any two saves": git merges non-adjacent hunks fine, so two frameworks updating existing rows usually merge. It bites when rows are *inserted* — two contributors each adding a framework, which is exactly when a rebase is most annoying. Reproduced both ways against the real data: old layout, actix + hyper update rows merges cleanly old layout, two new frameworks added CONFLICT in baseline-512 and -4096 new layout, the same two new frameworks merges cleanly, both files present Rows are copied verbatim rather than reshaped, which makes the migration provable: regenerating data.js before and after gives a byte-identical file. That is the main guarantee here — the board, the composite and every archived round see exactly what they saw before. Consumers: rebuild_site_data.py writes each framework's file; a run touches only the frameworks it benchmarked (verified: benchmarking actix + hyper modified 2 of 108 files) gen_leaderboard_data loads site/data/results/*.json once and rebuilds the per-profile view, sorted by framework name so the emitted data.js does not churn compare.sh one read of the framework's own file instead of a scan of every profile file archive.sh regroups into the existing round shape, so rounds/*.json keeps its current format benchmark-pr.yml stages only site/data/results/<slug>.json The filename is the display_name lowercased with anything outside [A-Za-z0-9._-] replaced by '-', because two gateway entries are named "aspnet-minimal + nginx" and "+ caddy". .gitignore's `results/` rule was unanchored and silently swallowed the new directory — the first attempt committed the code and none of the data. It is now `/results/`, which is the raw benchmark output it was always meant to cover; the root directory is still ignored. scripts/migrate_results_layout.py is kept rather than deleted: it re-derives the layout from a flat checkout, which is what a branch predating this needs in order to rebase. Note for #1020, which edits rebuild_site_data.py and deletes rows from the flat files: it will need a rebase, and running the migration script after merging main is the mechanical way to do it.
This was referenced Jul 24, 2026
MDA2AV
added a commit
that referenced
this pull request
Jul 24, 2026
#1034 added `env: FRAMEWORK` to the "Commit saved results" step, which already declared `env: GH_TOKEN` at its end. Two env keys in one step is invalid, and GitHub rejects the entire file: failed to parse workflow: (Line: 160, Col: 9): 'env' is already defined So /benchmark is currently dead on main — the workflow cannot be dispatched at all. Merged into the existing block. This slipped through because the check I ran was `yaml.safe_load`, and PyYAML silently accepts duplicate mapping keys (last one wins) while GitHub Actions does not. All six workflows now validate with a loader that raises on duplicates instead.
MDA2AV
added a commit
to Kaliumhexacyanoferrat/HttpArena
that referenced
this pull request
Jul 24, 2026
Six merges landed since the last sync: MDA2AV#1030, MDA2AV#1031, MDA2AV#1032, MDA2AV#1033, MDA2AV#1034 and MDA2AV#1036. One real conflict, in the composite blurb, where both sides had changed the same line for different reasons: main (MDA2AV#1032) made the text state which normalization rule is active, so a screenshot can't be ambiguous about whether scores were rescaled to the current filter this branch pointed the "How it works" link at /docs/scoring/composite -score/ instead of the hash route Both are kept: MDA2AV#1032's wording with this branch's real URL. Everything else merged on its own, but the two structural ones were checked rather than assumed: MDA2AV#1034 moved results from 52 shared per-profile arrays to one file per framework. Git applied that to the renamed generator correctly — it still loads site/data/results/*.json and still emits 1941 rows, and compare.sh kept both its per-framework read and the --compare flag from MDA2AV#1031. MDA2AV#1030 added three echo-ws-limited doc pages after this branch's SEO pass, so they were written with seo_title/description already; all 126 pages now carry both. Generator output on the merged tree: 1941 rows, 126 static doc pages, a 126 -page search index and a 127-URL sitemap. All six workflows validate against a loader that rejects duplicate keys, which is what MDA2AV#1036 had to fix.
MDA2AV
added a commit
to prothegee/HttpArena
that referenced
this pull request
Jul 24, 2026
This branch saved results before MDA2AV#1034, so it carries site/data/<profile>-<conns>.json files that main has since replaced with one file per framework. Git reported them as modify/delete conflicts. The rows this PR added or changed against its merge base were re-applied to site/data/results/zix.json and the flat files dropped. Only rows the PR actually touched moved across, so nothing main has changed since is resurrected: baseline-h3-64 9,573,685 -> 9,392,865 rps static-h3-64 611,890 -> 686,550 rps
MDA2AV
added a commit
to prothegee/HttpArena
that referenced
this pull request
Jul 24, 2026
This branch saved results before MDA2AV#1034, so it carries site/data/<profile>-<conns>.json files that main has since replaced with one file per framework; git reported them as modify/delete conflicts. The six gRPC rows this PR added were re-applied to site/data/results/zix.json and the flat files dropped: stream-grpc-64, stream-grpc-tls-64, unary-grpc-256, unary-grpc-1024, unary-grpc-tls-256, unary-grpc-tls-1024 Only rows the PR actually touched moved across, so nothing main has changed since is resurrected. site/data/frameworks.json also conflicted. It is derived from every frameworks/*/meta.json, so it was regenerated with rebuild_site_data.py::rebuild_frameworks_json rather than hand-merged - 118 entries, this PR's zix entry included.
MDA2AV
added a commit
to prothegee/HttpArena
that referenced
this pull request
Jul 24, 2026
This branch saved results before MDA2AV#1034, so it carries 20 site/data/<profile>-<conns>.json files that main has since replaced with one file per framework; git reported them all as modify/delete conflicts. Every row this PR added or changed against its merge base was re-applied to site/data/results/zix.json and the flat files dropped - 8 profiles this entry did not publish before (api-4, api-16, async-db, crud, json-comp at three connection counts, json-tls) and 12 it updated (baseline, limited-conn, pipelined, json, static, upload). Two keys already in main were left alone, because this PR did not touch them.
MDA2AV
added a commit
to williamthome/HttpArena
that referenced
this pull request
Jul 24, 2026
This branch saved results before MDA2AV#1034, so it carries 38 site/data/<profile>-<conns>.json files that main has since replaced with one file per framework; git reported them all as modify/delete conflicts. Every row this PR re-measured for the v0.8.0 bump was re-applied to site/data/results/roadrunner.json and the flat files dropped. All 38 keys already existed, so all 38 are updates rather than additions - for example baseline-512 833,311 -> 919,625 rps and async-db-1024 84,434 -> 134,059 rps. Only rows this PR touched moved across, so nothing main has changed since is resurrected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #751.
Results lived in 52 shared arrays —
site/data/<profile>-<conns>.json, each holding every framework — so a saved run rewrote files every other framework also occupies. They now live one file per framework:108 files, 1941 rows, rows copied verbatim.
The conflict is real, but narrower than "any two saves"
Git merges non-adjacent hunks fine, so I checked rather than assumed. Reproduced all three cases against the real data with actual
git merge:actix+hyperupdate existing rowsbaseline-512+baseline-4096It bites when rows are inserted — which is exactly the case where a rebase is most annoying, and the one the issue describes.
The correctness guarantee
Rows are copied verbatim rather than reshaped, so the migration is provable: regenerating
data.jsbefore and after produces a byte-identical file. The board, the composite and every archived round see exactly what they saw before.Consumers
rebuild_site_data.pysite/data/results/*.jsononce and rebuilds the per-profile view, sorted by framework name sodata.jsdoesn't churncompare.sharchive.shrounds/*.jsonkeeps its current formatbenchmark-pr.ymlsite/data/results/<slug>.jsonFilenames are the
display_namelowercased with anything outside[A-Za-z0-9._-]replaced by-, because two gateway entries are namedaspnet-minimal + nginxand+ caddy.One trap worth flagging
.gitignore'sresults/rule was unanchored, so it silently swallowedsite/data/results/— my first commit contained all the code and none of the data. It's now/results/, matching the raw benchmark output it was always meant to cover; the root directory is still ignored (verified both ways).Sequencing with #1020
#1020 edits
rebuild_site_data.pyand deletes rows from the flat files, so it will need a rebase.scripts/migrate_results_layout.pyis kept in the tree for exactly that: re-run it after merging main and the purge carries over mechanically. Happy to do that rebase — say which order you'd like these to land.🤖 Generated with Claude Code