[aniravich]

Summary

vllm-router ends its startup wait as soon as one worker is healthy, and reads each worker's model only at that moment. Workers that aren't up yet are filed under the placeholder model unknown, and requests that name the real model skip them until the router's periodic model refresh (every 5th health check, 240 s after startup on v0.2.1). The launcher started the router alongside the engines, so engines that came up later sat idle while sticky sessions pinned to the early ones.

On a Qwen3.8-27B SWE run with 16 vLLM servers, the router came up with 2 of 16 servers healthy, and all 335 opening sessions of the step-0 eval landed on those 2. They filled their KV cache and preempted constantly, the concurrency controller throttled the whole pool, 296 of 500 eval rollouts failed (59.2%), and the first training step took 1 h 29 min. Two earlier smoke runs of the same config hit the same race, with 10 of 16 and 1 of 16 servers listed when the router started.

launch_router now waits until every worker in the router's worker args answers /health, then execs vllm-router, so its one-time model discovery sees all of them. exec keeps the router as the background job, so a router crash still tears the job down, and a worker that never becomes healthy fails the launch after the same 4200 s timeout as before. This covers the RL, SFT and inference templates in regular and P/D mode; llm-d is unchanged. It also corrects the comment in clients.py that said the router only serves once its workers are healthy.

Companion router fix: PrimeIntellect-ai/router#57 (late workers join their model's pool on the next health check instead of the 5th). Once it's released, bump the vllm-router pin in pyproject.toml.

Test plan

• Dry run of the Qwen3.8-27B SWE config on this branch; bash -n on the rendered sbatch and on the inner srun launch script; tests/unit/orchestrator/test_clients.py passes.
• Rendered every multi-node template (RL, SFT, inference; regular and P/D; vllm-router and llm-d): all scripts and inner srun bodies pass bash -n, and the llm-d renders are byte-identical to main.
• The gate against fake workers, one binding 12 s late and one answering 503 for 8 s: the router starts only once both are healthy, as the same background PID. A worker that never starts makes the gate exit 1 without starting the router. Under bash 5.2, the template's wait -n -p teardown fires on both a gate timeout and a router crash.
• The failing config relaunched with this change and the patched router (6 nodes, 16 servers):

	Before	After
First to last server ready	2 min 52 s	4 min 15 s
Servers listed when the router started	2 of 16	16 of 16
First 335 sessions	167 / 168 on 2 servers	20–21 on each of 16
Load during the eval	2 servers at 99–100% KV with 40–46 waiting (1,014 preemptions); 14 idle	15–21 running on every server, 0 waiting, KV 12–18%, 0 preemptions
All 500 eval rollouts dispatched	68 min	12 min
Step-0 eval time	2 h 4 min	1 h 9 min
Step-0 eval errors	59.2% (296 / 500)	6.2% (31 / 500)
Step-0 eval solved (of all 500)	34.4%	78.2%
First training step	1 h 29 min	29 min

In the relaunch, the gate logged All 16 workers healthy after 592s, the router logged 16 out of 16 unique hosts are healthy with 16 Discovered model lines, and nothing was re-indexed later.