feat(explore): route EXPLORE (and SWEEP) through InferaSim instead of booting every candidate - #1494
Open
araina-amd wants to merge 14 commits into
Open
araina-amd wants to merge 14 commits into
araina-amd wants to merge 14 commits into
Conversation
araina-amd
requested review from
a team,
Ahmedhasssan-aig,
chao-xu-spec,
devalshahamd,
haofrank,
iraj465,
tsrikris and
yueliu14
as code owners
September 11, 2026 20:04
Projects a candidate's serving metrics analytically and returns the same benchmark_report.json contract, so real GPUs are only needed where the kernel regime changes. An anchor is reused across transport axes (TP/EP/PP, batch, concurrency, sequence lengths); a dtype or attention-backend change is flagged as needing a fresh one. Real-weights anchors rank ahead of dummy-weight ones, whose synthetic MoE routing flattens the decode curve: projected TPOT 30.7% -> 2.4% MAPE against measured serving for gpt-oss-120B. Co-authored-by: Cursor <cursoragent@cursor.com>
Three ways the backend could answer confidently from an anchor that did not describe the candidate. Speculative decoding was not parsed, so every mtp variant inherited a plain-decode anchor -- worth 33-91% on DeepSeek-R1. All three framework spellings now land on the regime axis. Corrupt anchors were selectable: decode latency cannot fall as batch rises, and 18 of 62 artifacts on this cluster fail that check, so they are rejected in favour of pure analysis. DeepSeek-R1, DeepSeek-V2-Lite and MiniMax-M2 had no preset mapping.
The gate reads whatever JSON sits in an anchor directory, and analysis output
lives alongside artifacts. A top-level list or scalar reached doc.get("sweep")
and raised AttributeError out of select_anchor instead of being rejected.
The filter compared spec.model_path, a checkout path, against the artifact's HuggingFace id. Those never match, so it selected nothing and then fell back to the full entry list -- any workload could calibrate against any anchor. Three different models returned the same decode step to two decimal places. Names now go through regime.models_match, shared with the Infera launcher rather than duplicated. An anchor with no recorded model is still allowed, since a structural warmup names no checkpoint.
A target arrives with a checkout path and a preset, and either one identifies it. Matching only on the preset rejected DeepSeek-R1's own anchor, since deepseek_v3 shares no spelling with its checkpoints; matching only on the path would miss a bare preset name. Both are now tried.
The first of each variant's two benchmarks exists to leave the server hot and run the accuracy gate; its throughput is discarded. It still runs at full length, five to ten waves of the concurrency. One wave reaches steady state. Across six boots and 54 seed-to-seed comparisons the first pass lands 0.97% faster than later ones -- the opposite of warming, and inside the run-to-run spread. Alongside it, a variant already benchmarked against this exact stack is not re-measured unless its verdict sits within two noise envelopes of the KEEP threshold. Gated by INFERENCE_OPTIMIZER_EXPLORE_SHORT_WARMUP; an unreadable config falls back to today's warmup. Co-authored-by: Cursor <cursoragent@cursor.com>
A kernel occupying f of GPU time and made S times faster cannot move end-to-end throughput past 1/((1-f) + f/S). When that ceiling is below the KEEP threshold, the integrate run boots a server to confirm arithmetic. The gate carries a 1.5x margin on the measured GPU share, so a mis-attributed profile does not discard a real win; unreadable share or speedup still runs the benchmark. Gated by HYPERLOOM_KERNEL_AMDAHL_GATE. Co-authored-by: Cursor <cursoragent@cursor.com>
The projection is validated against real vLLM measurements out to 65,536 tokens of context and never across a node boundary, but past either line it still returns a confident number -- the last such extrapolation hid a 46.3% error until the sweep was run. Projections now carry notes naming the axis being extrapolated along. These are deliberately not error bars: the size of the error is the unknown. Co-authored-by: Cursor <cursoragent@cursor.com>
_explore_screen.py adds an optional reduced-scale probe that can drop variants before the round benchmarks them on the deployment. Off by default, it only prunes, and prunes nothing unless probe and deployment agree on the kernels they resolved -- the same flags do not give the two engines the same stack. The EXPLORE plateau exit could not fire: its gain arm summed over winners_history, where every entry had already cleared KEEP, and its streak arm counted rounds that proposed nothing rather than rounds that kept nothing. Gated by HYPERLOOM_EXPLORE_PLATEAU_ROUND_WINDOW. Co-authored-by: Cursor <cursoragent@cursor.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Magpie's client-only path is env-driven, but Hyperloom emitted that env only for multi-node runs, so a single-node run could not target an externally hosted engine -- an Infera router fronting vLLM/SGLang, say. The gate was is_multi_node(), standing in for whether the server is ours to manage; uses_external_server() asks that directly, keyed on HYPERLOOM_MN_EXT_SERVICE_URL. _kill_stale_servers, recover's kill stage and both server_lifecycle eligibility checks now decline, since each would otherwise reap or reuse the very server being measured. Verified at unit level only. Co-authored-by: Cursor <cursoragent@cursor.com>
Match the InferaSim CLI and the INFERASIM_* env prefix. The selector is now HYPERLOOM_BENCHMARK_BACKEND=inferasim. Co-authored-by: Cursor <cursoragent@cursor.com>
araina-amd
force-pushed
the
araina/dev/infersim-benchmark-backend
branch
from
September 11, 2026 20:25
91fc101 to
e51c3de
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
The short-warmup path still named run_gv from the pre-rebase tree, so ruff F821 failed and the explore executor tests never reached Magpie. Also runs ruff format over the files this branch adds, and clears the CodeQL notes (unclosed files, duplicate json imports, unused constants). Co-authored-by: Cursor <cursoragent@cursor.com>
InferaSim made the engine a regime axis, because one checkpoint's vLLM and SGLang anchors were hashing to a single signature. We build the recipe ourselves and did not set it, and an axis missing on either side scores as matching, so a vLLM anchor and an SGLang anchor both came back at distance 0 -- the mismatch the branch already rejects for dtype and attention backend. The spec has carried the framework all along. Co-authored-by: Cursor <cursoragent@cursor.com>
select_anchor was the only thing keeping a vLLM measurement away from an SGLang candidate: the projection never named its engine, so Infera's own regime gate saw the axis as unset and matched as before. Passing --serving-engine closes that on the --load-benchmark path, where calibration is what the anchor is being read for. Nothing in the analytical model reads it, so no projected number moves. An unnamed framework still omits the flag rather than sending a blank. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wire InferaSim into Hyperloom as the EXPLORE benchmark backend so candidate configs are projected instead of standing up a full serving engine for every point. EXPLORE currently boots a real server per candidate; across the 9 shipped workloads that is 491 GPU-hours per search, almost all of it engine init rather than the benchmark. InferaSim already answers the same question (TTFT / TPOT / throughput / KV fit) from a short serving anchor plus an analytical transport.
The path is intended to be opt-in:
HYPERLOOM_BENCHMARK_BACKEND=inferasim
pointed at the warmup/anchor store. Magpie remains the default; this does not change FRAMEWORK_AGENT or KERNEL_AGENT (those change kernels, so each candidate is its own regime and projection saves nothing).
Why
From one real candidate artifact (
openai/gpt-oss-120b, TP=8, ISL 1024 / OSL 128, conc 64): 421 s wall, of which engine init (memory profiling, KV alloc, CUDA-graph capture) dominates. Weight load is ~4.5 s. Changing TP/EP/PP pays that again. 60 candidates × 8 GPUs × 421 s ≈ 56 GPU-hours for one workload.Those 60 configs differ in batch, parallelism and concurrency — exactly the axes InferaSim transports from one measured decode-step ladder.
What's included
HYPERLOOM_BENCHMARK_BACKEND=inferasimon the same benchmark call the grid runner already makes (today: Magpie /bypass).min(TP, 4)GPUs (~150 s boot + 72 s client = 222 s on 4 GPUs). Measured decode steps are the projection level; the model supplies deltas for other batch / TP / concurrency. For Arbor's shipped grid that is 8 anchors, not 60 servers.maxseqs, chunked-prefill) reuse an existing anchor; kernel levers (aiter*, attention backend, cudagraph, KV dtype) require their own.Validation (from the InferaSim-in-Hyperloom measurement)
Held-out decode step vs a real server, 40 points / 2 models:
TP8 gpt-oss-120b projected from a TP=4 anchor: c8 6.13 vs 6.04 ms (−1.5%), c32 7.21 vs 7.01 ms (−2.9%) — the 2.2% “against a real server” figure.
GPU-hours on Hyperloom's 9 shipped workloads (60-candidate EXPLORE grid):
Of the remaining 39.3 GPU-h, 24.5 are the three real validation runs per workload and 14.8 are anchors. Costless anchors would be 20× — that is the ceiling, set by how much real measurement we keep, not by the projector.
Attribution (pooled): reuse one anchor per regime is 5.73× / 94% of the saving; reduced-parallelism anchors +1.71×; short sweep vs full run +1.34×. Projecting without reuse saves nothing.
Notes / limitations
Test plan
HYPERLOOM_BENCHMARK_BACKENDstill uses MagpieHYPERLOOM_BENCHMARK_BACKEND=inferasimwith an empty anchor store harvests one short serving anchor per regime, then scores the rest of EXPLORE without a full engine bootmaxseqs, chunked-prefill) reuse the regime anchor