EP MoE generate fails after a successful tp=4 load: 'EP arch mismatch (expected dense Qwen TP)' (ornith-1.5:35b-a3b-mq4r, master)

[Kaden-Schutt]

Found by the hw-gate deciding seat while investigating #682, reproduced on master 344e17a2e and on #682 alike, so it is pre-existing.

Repro (hiptrx, 4× gfx1201, ROCm 7.15):

• hipfire serve 127.0.0.1 11520 --tp 4 with ornith-1.5:35b-a3b-mq4r (registry sha 84103fcc…), or the native protocol load with params.tp=4.
• Load succeeds: [loader] EP load complete: 4 ranks, diag reports model_loaded=true, arch qwen3_5_moe, ~16 s.
• Every generate fails: EP arch mismatch (expected dense Qwen TP) — 5/5 battery turns empty through serve_harness.py --tp 4, same through the raw daemon protocol (ornith_ep_tp4.BASE.json in the evidence set).

So the EP loader admits and builds the A3B MoE at tp=4, but the generate dispatch treats the loaded model as a dense Qwen TP model and refuses. Either the EP generate route for arch 6 (qwen3_5_moe) is not wired for this artifact, or the loaded state is mis-tagged.

Not touched by #682 (both binaries fail identically). Single-device generation on the same artifact is fine (206 tok/s AR). Evidence will be attached from the gate once #679 lands.

[Kaden-Schutt]

Traced. Not a clobber from the device-mesh series — a gap that has existed since Qwen3.5 EP was introduced, with #662 changing only the error string:

• e3c7b4c35 (2026-08-09, "add safe EP batch foundations") gave the loader EpArch::Qwen35 { batch, .. } and the arch crate ep_batch.rs::forward_ep, but hipfire-generate never got a generate arm for it — the only EpArch::Qwen35 match in the engine is the reset path. Before perf(qwen35): scale dense MQV2 tensor parallelism through TP5 #662, arch 6 EP fell through to ep_serve_ds4, which refused with EP arch mismatch (expected ds4).
• 616336932 (perf(qwen35): scale dense MQV2 tensor parallelism through TP5 #662, "scale dense MQV2 tensor parallelism through TP5") added ep_serve_qwen35_dense_tp and routed 5 | 6 => to it (crates/hipfire-generate/src/qwen.rs:240). Arch 6 is the MoE; the dense server pattern-matches EpArch::Qwen35DenseTp and refuses with the current message.

Net: the loader admits an A3B MoE at tp=4, spends ~16 s and ~19 GB across four GPUs building it, and the engine has no route to run it. Two acceptable fixes, either is a one-PR change:

1. Wire it: 6 => ep_serve_qwen35_moe(...) driving hipfire_arch_qwen35::qwen35::ep_batch::forward_ep over EpArch::Qwen35 { batch }, mirroring ep_serve_qwen35_dense_tp's prefill/decode/terminal structure.
2. Refuse at admission until (1) exists: load_model_ep_with_kv_mode arm 6 => returns a validation error before any allocation, and #662's dispatch becomes 5 => only.

(2) is the honest minimum for master today; (1) is what the A3B artifacts on 4× gfx1201 are waiting for.

[Kaden-Schutt]

@fivetide — flagging this for you since it sits squarely on the EP admission/dispatch seam that #682 reworks, and #682 is the natural place for at least the admission-side fix.

Concretely, in #682's admit_source (crates/hipfire-loader/src/admission.rs), the EP branch admits arch_id 6 and then the loader builds a 4-rank A3B that the engine cannot serve. Refusing 6 at admission (EP generation for Qwen3.5-MoE is not wired; load with tp=1) would make the transactional refusal path you built do the honest thing here — no 19 GB allocation for a model that cannot generate — and is a two-line change on top of the vmm fix from the PR comment. Wiring ep_serve_qwen35_moe over ep_batch::forward_ep is the real feature and can be its own PR after.

Not blocking #682 on the wire-up; the admission refusal (or an explicit note that you're leaving arch 6 EP as-is) is what I'd want to see before it lands.