Skip to content

[Clustered] Mixed pod shapes in one gang, e.g. a model server and its clients #8396

Open

Description

@mitja-kleider

We evaluate LLM checkpoints with one GPU pod serving the model (vLLM) and several CPU pods running benchmarks against it, each benchmark in its own image. We want the clustered plugin to run them as one gang:

  • admitted together: Kueue admits the server and its clients as one workload, so the GPUs don't sit idle while the CPU pods queue, and the clients don't start before the server can be admitted
  • evicted and resumed together: with resume_after_preemption, a preemption suspends the whole set, and re-admission resumes only the unfinished clients
  • torn down together: the set succeeds once the clients complete, which stops the server

The plugin builds a single ReplicatedJob (workers), so today the server and clients have to be separate tasks, which Kueue admits and evicts independently.

We validated this JobSet shape on our cluster outside Flyte (JobSet v0.12, Kueue v0.17): one vllm job plus one job per benchmark. Kueue admitted the mixed set as one workload. Deactivating the workload evicted all of it, and re-activating it resumed only the unfinished jobs. A SuccessPolicy over the benchmark jobs ended the set and removed vLLM.

A possible shape: ClusteredTaskSpec gains named roles, each its own ReplicatedJob with image, command, resources, pod template and replica count, and the SuccessPolicy targets a chosen subset of roles. Roles reach each other through the JobSet's headless service, e.g. <jobset>-vllm-0-0.<subdomain>.

Open questions:

  • Which role runs the task body and produces its outputs: the existing workers, or any role?
  • Failure semantics per role: JobSet v0.12 has no per-job fail action, so a failed client restarts the whole set unless the client itself exits 0.
  • SDK surface, e.g. extra roles on ClusteredTaskEnvironment as container specs.

Would you be open to this? We'd be happy to contribute it.

Activity

  1. runllm commented on Nov 12, 2026

    @runllm

    It seems likely this is a reasonable feature proposal, but the provided sources don’t establish whether the Flyte team would accept it or whether the current clustered plugin supports multiple named roles. They describe ClusteredTaskEnvironment as emitting a JobSet and the existing configuration as a single replicas/nproc_per_node worker shape—not the proposed per-role jobs, images, resources, commands, or success policy (ClusteredTaskEnvironment API).

    The sources also confirm that resume_after_preemption handles preemption for a clustered task, but don’t specify mixed-role resumption, per-role failure semantics, or which role would own task outputs. Those points would need design discussion; the cited docs can’t answer them. Would you like help turning this into a concise feature-request comment?

    Sources:


    This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

    Mention @Herald to ask follow-up questions. See runllm.com for more!

  2. AdilFayyaz commented on Nov 17, 2026

    @AdilFayyaz
    Contributor

    @mitja-kleider Thanks for writing this up and testing it outside Flyte first. Supporting different pod roles in one clustered task makes sense to me. I'm less sure about putting the benchmark clients in the same gang as the server, though.

    GKE's Qwen3 multi-host vLLM tutorial is a good comparison. It runs prefill, decode and proxy as roles in one JobSet, but the benchmark is a separate Job that waits for the server to be healthy and then runs.

    With the clients in the gang, one failing benchmark restarts vLLM too, since JobSet can't restart a single job. The benchmark scores also can't come back as task outputs easily, because only one role runs the task code.

    Could you run vLLM as one task and the benchmarks as separate tasks, with the parent stopping the server when they're done? If that doesn't work for you, what's missing? Is it idle GPUs while the clients queue, the clients not pausing when vLLM gets preempted, or something else?

  3. mitja-kleider commented on Nov 18, 2026

    @mitja-kleider
    ContributorAuthor

    Thanks, good questions. Separate tasks are what we have on Flyte 1 today, and the gaps are the ones you listed:

    • Idle GPUs. vLLM and its clients become separate Kueue workloads. vLLM gets admitted and holds its GPUs while the clients may wait for CPU quota behind other work. Or the clients start first and wait for a server that is still queued.
    • Preemption. In a preemptible queue, Kueue evicts vLLM alone. The clients don't pause. They fail or burn their retries against a dead server, and the parent has to coordinate the restart.
    • Teardown. If the parent stops the server, the parent has to stay alive for the whole eval. When the parent dies, its server still holds GPUs until something notices.

    Holding vLLM and its clients as one workload removes all three. That's the main thing we want from Flyte here.

    On your concerns:

    • A failing client restarts vLLM. Our clients are already wrapped to exit 0 and record a failure in their output, so a client never fails the JobSet. A per-job failure action in JobSet would be cleaner, but we don't need it.
    • Outputs. Our clients write scores to shared storage. workers could stay the only role that runs task code: its function waits for the clients' results, returns them as task outputs, and its success ends the set. vLLM and the clients would be plain image-and-command roles.

    I think the GKE tutorial is a different setup: it is running one benchmark against a server that will not be preempted during the benchmark. Ours is many short evals per checkpoint, each checkpoint running as a new server living only as long as its benchmarks, sharing a preemptible queue with training.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions