Repository navigation
fix(executor): apply TaskMetadata.metadata to plugin objects - #8406
Open
m86hussein wants to merge 1 commit into
Open
m86hussein wants to merge 1 commit into
m86hussein wants to merge 1 commit into
Conversation
TaskMetadata.metadata is meant for the task's pod or, for CRD-backed tasks, the CR itself, while K8sPod metadata goes to the CR's pod templates. v1 propeller merges it in addObjectMetadata (flyteorg#6421); the v2 executor only merged the default and execution labels and annotations, so there was no way to annotate a CR without annotating its pods too. That breaks Kueue elastic RayJobs, whose kueue.x-k8s.io/elastic-job annotation Kueue rejects on pods. Read the task template in launchResource and getResource and merge its metadata like v1 does: over the platform defaults and whatever the plugin put on the object, under the execution's own labels and annotations. Related to flyteorg#8392 Signed-off-by: m86hussein <m86hussein@gmail.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
A malformed inline template can block cleanup resource lookup, while Abort and Finalize treat the error as success, allowing the TaskAction finalizer to be removed without deleting the workload or clearing its finalizers.
1 open issue
What changed in this PR
Adds executor-side support for task-level labels and annotations on Kubernetes plugin objects, addressing #8392’s need to annotate a RayJob independently of its pods. SDK support remains separate.
Changes:
- Merges task metadata above defaults and plugin-provided metadata, while preserving execution metadata precedence.
- Reads task templates during resource creation and identity lookup.
- Adds precedence tests and verifies that task metadata cannot override the managed label.
| File | Description |
|---|---|
executor/pkg/plugin/k8s/plugin_manager.go |
Applies task-level metadata to plugin objects. |
executor/pkg/plugin/k8s/plugin_manager_test.go |
Tests metadata precedence and updates task-reader mocks. |
Review effort: Balanced
Comment on lines
+224
to
+229
| taskTemplate, err := tCtx.TaskReader().Read(ctx) | ||
| if err != nil { | ||
| logger.Errorf(ctx, "Failed to read task template. Error: %v", err) | ||
| return nil, err | ||
| } | ||
| pm.addObjectMetadata(tCtx.TaskExecutionMetadata(), o, config.GetK8sPluginConfig(), taskTemplate) |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Tracking issue
Related to #8392
This is the executor side. The SDK doesn't set
TaskMetadata.metadatayet either; that needs a small flyte-sdk change (labels/annotations on the task, like flytekit's@task(labels=..., annotations=...)), which I'll send separately.Why are the changes needed?
TaskMetadata.metadatais meant for the task's pod or, for CRD-backed tasks, the CR itself, whileK8sPodmetadata goes to the CR's pod templates (see the comment on the field inflyteidl2/core/tasks.proto). v1 propeller merges it inaddObjectMetadata(#6421), but the v2 executor'saddObjectMetadataonly merges the default and execution labels and annotations.So in v2 there's no way to annotate a CR without also annotating its pods. That breaks Kueue elastic RayJobs: the RayJob needs
kueue.x-k8s.io/elastic-job: "true", and Kueue rejects pods that carry it.What changes were proposed in this pull request?
launchResourceandgetResourceread the task template and pass it toaddObjectMetadata, which mergesTaskMetadata.metadatathe same way v1 does. Precedence, lowest to highest:K8sPluginConfigdefault labels/annotationsTaskMetadata.metadataTaskExecutionMetadata, which also carries the managed label)So task-level metadata overrides what a plugin derived from the pod template, but can't override the execution's own labels, including the label the executor's pod cache selects on.
How was this patch tested?
TestAddObjectMetadata_TaskMetadatachecks the precedence above for labels and annotations, including a task annotation overriding a pod-template one on the object.TestAddObjectMetadata_ManagedLabelnow also checks that task metadata can't override the managed label.TaskReader.go test ./executor/pkg/plugin/...andgo vet ./executor/pkg/plugin/k8s/...pass locally.Labels
Check all the applicable boxes