[garciadias]

Description

flClient.gpu.enabled=true did not allocate a GPU through Kubernetes. It rendered only an nvidia.com/gpu annotation, a toleration and NVIDIA_VISIBLE_DEVICES=0, with no resource request. That only works where the node's default container runtime is nvidia, and even there the pod takes physical GPU 0, which nothing reserves. On a cluster whose GPUs come from the NVIDIA device plugin or the GPU Operator (the setup the chart docs require), the client could be scheduled onto a node with no GPU, or share GPU 0 with another workload. That is the case for the GSTT trust in #1766, which runs on an 8-GPU cluster.

This PR adds flClient.gpu.allocation:

• devicePlugin (default): the fl-client container requests nvidia.com/gpu: <count>, merged into flClient.resources.limits the way monailabel already does it. The device plugin picks the device and sets NVIDIA_VISIBLE_DEVICES itself, so the chart no longer sets it; a value in the pod spec would override the plugin's choice. CUDA_VISIBLE_DEVICES=0 stays, since the pod's one GPU is index 0 inside the container.
• runtime: the previous shape, with no request and NVIDIA_VISIBLE_DEVICES=0. It is for a single node with no device plugin and default-runtime: nvidia, for example a k3s workstation.

An unknown value is refused, both by values.schema.json and by the template, which covers a render without schema validation.

The docs (chart README, troubleshooting list and the Kubernetes trust page) also gain the symptom from #1766: a trust left at flClient.gpu.enabled=false offers FLARE 0 GPUs, so every job that asks for one waits and then fails. They point multi-GPU clusters at NVFLARE job pods (flClient.nvflare.jobPods), which allocate a GPU per job.

Linked Issues

Relates to #1766. That issue also covers operational work on the GSTT cluster that this PR does not do.

Checklist

• Follows the project's coding conventions and style guide
• Updates documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published

Type of Change

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• In-line docstrings updated.
• Documentation updated, tested make -C docs/ docs.

Upgrade note: a release with flClient.gpu.enabled=true on a node that has no device plugin (single-node k3s relying on default-runtime: nvidia) must add flClient.gpu.allocation: runtime before upgrading. Without it the fl-client requests nvidia.com/gpu, which that node does not advertise, and the pod stays Pending. Releases with the GPU disabled are unaffected.

Testing

I did the following tests to verify my changes:

• tests/templates/test_fl_client_gpu_render.py (new) fails on develop and passes here. It covers:
  • the default request and that NVIDIA_VISIBLE_DEVICES is left to the plugin;
  • an operator's own limits are kept;
  • the runtime shape;
  • a CPU-only client gets no GPU;
  • an unknown value is refused.
• The whole chart suite: uv run --no-project --with pytest --with pyyaml pytest trust/deploy/helm/tests (538 passed).
• make -C trust/deploy/helm test (lint, template, validate), plus kubeconform -strict over a GPU-enabled render (56 resources valid).
• ruff check / ruff format --check on the new test.
• Integration tests: not applicable.

Additional Notes

Job-pod mode (flClient.nvflare.jobPods) already requests nvidia.com/gpu per job and is unchanged.