fix(bench): decode-history graph plotted prefill runs as decode points - #104
Merged
Conversation
/v1/history filtered on model_id and lane only, but a model normally has pp (prefill) records alongside its tg (decode) ones under the same model_id and lane — and a pp record carries no decode figure at all. The run drawer's "decode history" sparkline was therefore plotting a series that was partly prefill runs. Measured against real CT105 data: MiniCPM5-1B returns 8 points for model+lane, only 4 of which belong on the graph. Different config_labels are the same apples-to-oranges problem — two configurations of a model, not successive measurements of one. The endpoint gains optional `kind` and `config` filters and the drawer pins both: `kind=tg`, and `config` to the row's own variant. Filters stay optional so the CLI-shaped ?cell_key= form is unaffected. Filtering by display dimensions rather than by cell_key is deliberate even though a cell is exactly "one comparable series": cell_key is content-addressed over engine/image provenance, so an unrelated runner-image bump between sweeps forks the key and shatters one continuous history into several one-point series. hal0's own dashboard moved off cell_key for this reason. cell_key remains available as an explicit filter. Missed in #63 — the endpoint's own tests seeded one record per model, so no test had a pp row to leak. Both filters are now covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Deploying hal0-web with
|
| Latest commit: |
0af5049
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://f02a986a.hal0-web.pages.dev |
| Branch Preview URL: | https://fix-bench-history-display-di.hal0-web.pages.dev |
thinmintdev
added a commit
that referenced
this pull request
Aug 10, 2026
Two comment blocks outlived #63 and #104 — normalizeHistoryPoints' docstring and the run-drawer graph note both described /v1/history as proposed and adapter-only, and the docstring listed the old display dims (kind/outcome hardcoded) rather than the kind/config the caller now pins. Comments only, no behaviour change. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
/v1/history(added in #63) filtered onmodel_idandlaneonly. A model normally has pp (prefill) records alongside its tg (decode) ones under the same model_id and lane, and a pp record carries no decode figure at all — so the run drawer's "decode history" sparkline was plotting a series that was partly prefill runs.Measured against real CT105 data:
Different
config_labels are the same apples-to-oranges problem: two configurations of a model, not successive measurements of one.The fix
/v1/historygains optionalkindandconfigfilters; the drawer pins both —kind=tg, andconfigto the row's own variant. Filters stay optional, so the CLI-shaped?cell_key=form is unaffected.Why display dimensions and not cell_key
A cell is exactly "one comparable series", so filtering by
cell_keylooks like the obvious answer. It isn't:cell_keyis content-addressed over engine/image provenance, so an unrelated runner-image bump between two sweeps forks the key and shatters one continuous history into several one-point series. hal0's own dashboard moved offcell_keyfor this reason. It stays available as an explicit filter for callers that genuinely want that one identity.How it got through
My miss in #63. The endpoint's tests seeded one record per model, so no test had a pp row available to leak into a series. I also saw the symptom during end-to-end verification — a first point of
{"decode_ts_med": null, "prefill_ts_med": 6951.18}— and read past it.Both filters are now covered by tests.
Verification
npm run buildkind:"tg"ships in the built island bundle, not just the source🤖 Generated with Claude Code