Skip to content

fix(executor): apply TaskMetadata.metadata to plugin objects - #8406

Open
m86hussein wants to merge 1 commit into
flyteorg:mainfrom
m86hussein:fix/executor-task-metadata-on-cr
Open

m86hussein wants to merge 1 commit into
flyteorg:mainfrom
m86hussein:fix/executor-task-metadata-on-cr

Conversation

@m86hussein

Copy link
Copy Markdown

Tracking issue

Related to #8392

This is the executor side. The SDK doesn't set TaskMetadata.metadata yet either; that needs a small flyte-sdk change (labels/annotations on the task, like flytekit's @task(labels=..., annotations=...)), which I'll send separately.

Why are the changes needed?

TaskMetadata.metadata is meant for the task's pod or, for CRD-backed tasks, the CR itself, while K8sPod metadata goes to the CR's pod templates (see the comment on the field in flyteidl2/core/tasks.proto). v1 propeller merges it in addObjectMetadata (#6421), but the v2 executor's addObjectMetadata only merges the default and execution labels and annotations.

So in v2 there's no way to annotate a CR without also annotating its pods. That breaks Kueue elastic RayJobs: the RayJob needs kueue.x-k8s.io/elastic-job: "true", and Kueue rejects pods that carry it.

What changes were proposed in this pull request?

launchResource and getResource read the task template and pass it to addObjectMetadata, which merges TaskMetadata.metadata the same way v1 does. Precedence, lowest to highest:

  1. K8sPluginConfig default labels/annotations
  2. whatever the plugin put on the object (e.g. pod template metadata a plugin copies onto its CR)
  3. TaskMetadata.metadata
  4. the execution's labels/annotations (TaskExecutionMetadata, which also carries the managed label)

So task-level metadata overrides what a plugin derived from the pod template, but can't override the execution's own labels, including the label the executor's pod cache selects on.

How was this patch tested?

  • New TestAddObjectMetadata_TaskMetadata checks the precedence above for labels and annotations, including a task annotation overriding a pod-template one on the object.
  • TestAddObjectMetadata_ManagedLabel now also checks that task metadata can't override the managed label.
  • Existing launch tests give their task context a TaskReader.
  • go test ./executor/pkg/plugin/... and go vet ./executor/pkg/plugin/k8s/... pass locally.

Labels

  • fixed

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

TaskMetadata.metadata is meant for the task's pod or, for CRD-backed tasks,
the CR itself, while K8sPod metadata goes to the CR's pod templates. v1
propeller merges it in addObjectMetadata (flyteorg#6421); the v2 executor only
merged the default and execution labels and annotations, so there was no
way to annotate a CR without annotating its pods too. That breaks Kueue
elastic RayJobs, whose kueue.x-k8s.io/elastic-job annotation Kueue rejects
on pods.

Read the task template in launchResource and getResource and merge its
metadata like v1 does: over the platform defaults and whatever the plugin
put on the object, under the execution's own labels and annotations.

Related to flyteorg#8392

Signed-off-by: m86hussein <m86hussein@gmail.com>
Copilot AI balanced review requested due to automatic review settings November 14, 2026 16:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A malformed inline template can block cleanup resource lookup, while Abort and Finalize treat the error as success, allowing the TaskAction finalizer to be removed without deleting the workload or clearing its finalizers.

1 open issue
What changed in this PR

Adds executor-side support for task-level labels and annotations on Kubernetes plugin objects, addressing #8392’s need to annotate a RayJob independently of its pods. SDK support remains separate.

Changes:

  • Merges task metadata above defaults and plugin-provided metadata, while preserving execution metadata precedence.
  • Reads task templates during resource creation and identity lookup.
  • Adds precedence tests and verifies that task metadata cannot override the managed label.
File Description
executor/​pkg/​plugin/​k8s/​plugin_manager.go Applies task-level metadata to plugin objects.
executor/​pkg/​plugin/​k8s/​plugin_manager_test.go Tests metadata precedence and updates task-reader mocks.

Review effort: Balanced

Comment on lines +224 to +229
taskTemplate, err := tCtx.TaskReader().Read(ctx)
if err != nil {
logger.Errorf(ctx, "Failed to read task template. Error: %v", err)
return nil, err
}
pm.addObjectMetadata(tCtx.TaskExecutionMetadata(), o, config.GetK8sPluginConfig(), taskTemplate)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants