Skip to content

[Flyte 8316] Apply default task resource to action and pod. - #8372

Open
0yukali0 wants to merge 3 commits into
flyteorg:mainfrom
0yukali0:flyte-8316
Open

0yukali0 wants to merge 3 commits into
flyteorg:mainfrom
0yukali0:flyte-8316

Conversation

@0yukali0

@0yukali0 0yukali0 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Tracking issue

Closed #8316

Why are the changes needed?

Project task_resource settings are resolved but never applied. To give admins resource guardrails,
the settings need to be turned into default task resources and carried through run creation, the
action, and the pod, so that tasks get default values and are capped at the configured max.

What changes were proposed in this pull request?

  • runs: CreateRun converts the resolved task_resource settings into task_resource_defaults.
  • actions: Adds default task resources to the TaskAction CR and passes in the defaults from CreateRun.
    Child actions inherit them from their parent.
  • executor plugin: Applies the default task resources and the default accelerator to the task pod.

How was this patch tested?

  1. make devbox-build && make devbox-run APPLE_SILICON=True # I tested on the Mac.
    [Optional] flyte get settings --domain development --project flytesnacks

  2. res.yaml

task_resource.min.cpu: "500m"
task_resource.min.memory: "512Mi"
task_resource.min.storage: "1Gi"
task_resource.max.cpu: "2"
task_resource.max.memory: "2Gi"
task_resource.default_accelerator: apple-silicon 
  1. flyte edit settings --domain development --project flytesnacks -f ~/res.yaml
  2. flyte get settings --domain development --project flytesnacks
Screenshot 2026-11-18 at 6 27 07 AM

Tasks

  1. Apple-silicon host mode which requires environment setting manually so it is only for test.
Screenshot 2026-11-18 at 6 29 25 AM
  1. run hello.py example
    2.1 no resource setting adopts min resource in setting then set the resource with min resources.
Screenshot 2026-11-18 at 7 09 43 AM 2.2 resource requirement is larger than setting then set the resource with max resources. Screenshot 2026-11-18 at 7 04 46 AM

Labels

  • added: For new features.

Setup process

Screenshots

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Signed-off-by: Yuteng Chen <a08h0283@gmail.com>
…askresource fields

Signed-off-by: Yuteng Chen <a08h0283@gmail.com>
…fault_accelerator

Signed-off-by: Yuteng Chen <a08h0283@gmail.com>
@0yukali0
0yukali0 marked this pull request as ready for review November 17, 2026 23:13
@0yukali0
0yukali0 marked this pull request as draft November 17, 2026 23:15
@0yukali0
0yukali0 marked this pull request as ready for review November 17, 2026 23:25
@Chiuuu0209

Copy link
Copy Markdown
Contributor

Tested this on Linux with an NVIDIA GPU (RTX 5080 Laptop, 16 GB), using the GPU devbox built from this branch (0da7eba, make build + Dockerfile.gpu, nvidia device plugin v0.19.1, k3s v1.34.6+k3s1) and flyte SDK 2.13.6.

Project settings (flytesnacks / development):

task_resource.max.gpu: "1"
task_resource.default_accelerator: nvidia-l4

The node is labeled k8s.amazonaws.com/accelerator=nvidia-l4 (default gpu-device-node-label).

Task resources nvidia.com/gpu (req / limit) Node affinity + toleration Result
gpu=1 1 / 1 nvidia-l4 (default applied) Succeeded, nvidia-smi sees the RTX 5080
gpu="T4:1" 1 / 1 nvidia-tesla-t4 (task's own kept) Pending: node didn't match affinity, as expected
gpu=2 1 / 1 (capped by max) nvidia-l4 Succeeded, nvidia-smi sees the RTX 5080
no GPU none none Succeeded, no nvidia.com/gpu allocated
child gpu=1 task called from a CPU-only parent parent: none, child: 1 / 1 parent: none, child: nvidia-l4 Succeeded, child sees the RTX 5080; both TaskAction CRs carry the same taskResourceDefaults (copied from the parent)
gpu=1 with default_accelerator: ~unset (control) 1 / 1 none Succeeded, no accelerator affinity or toleration

So once the node label matches, the pod does get scheduled onto the GPU node and sees the GPU. The nodeAffinity on the gpu=1 pod comes from this PR (without default_accelerator there is none), and the gpu=2 pod would stay Pending on this 1-GPU node without the cap.

One note: default_accelerator only accepts AcceleratorModel names, so a consumer card like the RTX 5080 can't be named directly, and I used nvidia-l4 as a stand-in label. Not a blocker for this PR, just something people running on non-cloud GPUs may hit.

I only set max.gpu and default_accelerator here; the CPU / memory min-max paths weren't re-tested on this box.

@0yukali0 0yukali0 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Tasks get the settings min as their default resource request, clamped so it never exceeds max, and max caps both the task's requests and limits.
  2. When the settings set a min for ephemeral storage, a task that doesn't set storage gets that value as its request and limit.
  3. The settings' default_accelerator is applied to tasks that request GPUs but don't name an accelerator.

}
}

func newTaskResourceDefaults(tr *settings.TaskResourceSettings) *task.TaskResourceDefaults {

@0yukali0 0yukali0 Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting providing min and max resources (called res1 and res2) sets the pod spec.resource.

  • Treat the res2 as big one and set the limit with it.
  • Set the request with res1 if res1 < res2. otherwise, set it with res2.

},
},
{
name: "INHERIT and UNSET dimensions is ignored",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolveSettings handles the merge of setting from the hierarchy including org, domain and project.
INHERIT(default) and UNSET(~unset) shows up in some scenarios.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[V2] Apply settings task resource min/max and default accelerator

2 participants