Repository navigation
IDL: action input and code overrides and a fork marker for recovery runs - #8234
Conversation
Recover gains action_input_overrides (action names + data-plane URI of ActionInputOverridesData) and server-set implied_rerun_actions. The actions Action and ActionMetadata carry input_override_applied, RunDetails reports per-override status, and CreateRunResponse carries warnings. Signed-off-by: Niels Bantilan <niels@union.ai> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A fork that only changes code carries no action input overrides, so clients could not tell it apart from a plain recover. Recover.fork records that the run is a fork; it is informational and does not change recovery. Also regenerates the Rust bindings, which missed the earlier action-input-overrides fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Add ActionInputOverridesData.code_bundles: action name -> ActionCodeBundle, the code bundle archive that one action of a fork runs in place of the one its parent passes down. The parent's controller applies it when it enqueues the action, like the action's input overrides, so a fork can patch the code of a single task action without renaming it or changing the rest of the run. The bundle URI lives in the overrides file in the data plane; the RunSpec still carries only action names and the file's location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
…nput-overrides # Conflicts: # gen/go/flyteidl2/actions/actions_service.pb.go # gen/python/flyteidl2/actions/actions_service_pb2.py # gen/rust/src/flyteidl2.actions.rs # gen/rust/src/flyteidl2.workflow.rs # gen/ts/flyteidl2/actions/actions_service_pb.ts
Review feedback: ActionInputOverridesData now maps each overridden action to the URI of a serialized Inputs in the data plane (input_uris), instead of embedding the Inputs. This matches how an action's inputs are referenced (input_uri), keeps the overrides file small however many overrides a fork carries, and lets the parent's controller read only the override inputs of the actions it launches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Only generated bindings conflicted; the .proto sources merged cleanly (main's queue-scoped ListActions, ActionStatus queue timestamps and TaskActionMetadata.queue sit beside the override fields). Regenerated gen/ with buf 1.58.0, the version gen.Dockerfile pins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
…nput-overrides Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> # Conflicts: # gen/go/flyteidl2/actions/actions_service.pb.go # gen/go/flyteidl2/task/run.pb.go # gen/go/flyteidl2/workflow/run_service.pb.go # gen/python/flyteidl2/actions/actions_service_pb2.py # gen/python/flyteidl2/task/run_pb2.py # gen/python/flyteidl2/workflow/run_service_pb2.py # gen/rust/src/flyteidl2.actions.rs # gen/rust/src/flyteidl2.task.rs # gen/rust/src/flyteidl2.workflow.rs # gen/ts/flyteidl2/actions/actions_service_pb.ts # gen/ts/flyteidl2/task/run_pb.ts # gen/ts/flyteidl2/workflow/run_service_pb.ts
There was a problem hiding this comment.
🟢 Approval recommended
The review identifies no active issues in the recovery-override interface changes.
No open issues 🎉
12 files not reviewed
gen/go/flyteidl2/actions/actions_service.pb.go: Generated filegen/go/flyteidl2/actions/actions_service.pb.validate.go: Generated filegen/go/flyteidl2/task/common.pb.go: Generated filegen/go/flyteidl2/task/common.pb.validate.go: Generated filegen/go/flyteidl2/task/run.pb.go: Generated filegen/go/flyteidl2/task/run.pb.validate.go: Generated filegen/go/flyteidl2/workflow/run_definition.pb.validate.go: Generated filegen/python/flyteidl2/actions/actions_service_pb2.py: Generated filegen/python/flyteidl2/task/common_pb2.py: Generated filegen/python/flyteidl2/task/run_pb2.py: Generated filegen/python/flyteidl2/workflow/run_definition_pb2.py: Generated filegen/python/flyteidl2/workflow/run_service_pb2.py: Generated file
Review effort: Balanced
SVilgelm
left a comment
There was a problem hiding this comment.
No major issues detected. Three small nits inline.
reviewed by Claude (Opus 5.5)
| // Input and code overrides for task actions of the run being recovered. Set by the | ||
| // client. Only the action names and the location of the overrides travel in the | ||
| // RunSpec; the values and code live in the data plane (see ActionInputOverridesData). | ||
| ActionInputOverrides action_input_overrides = 2; |
There was a problem hiding this comment.
Nit: ActionInputOverrides, ActionInputOverridesData and input_override_applied now also carry code overrides (code_bundles). A neutral name (e.g. ActionOverrides, override_applied) would be cheap now and is permanent once released.
There was a problem hiding this comment.
Renamed in 3eef58e: ActionOverrides, ActionOverridesData, ActionOverrideStatus, Recover.action_overrides, RunDetails.action_override_status and override_applied. Field numbers are unchanged. flyte-sdk#1708, unionai/cloud#19498 and unionai/flyteplugins-union#204 are updated to match.
| // that creates the fork (for example a code-only fork); the server also sets it | ||
| // whenever action_input_overrides is present, and clears it on a run that is not | ||
| // a recovery. Informational: it does not change how the run is recovered. | ||
| bool fork = 4; |
There was a problem hiding this comment.
Nit: "clears it on a run that is not a recovery". The Union backend rejects any recover on a non-recovery run (InvalidArgument), so it never gets cleared.
There was a problem hiding this comment.
Fixed in 4407aed: the comment no longer says the server clears it.
| repeated string action_names = 1 [(buf.validate.field).repeated.items.string.min_len = 1]; | ||
|
|
||
| // Location of a serialized ActionInputOverridesData in the data plane. | ||
| string uri = 2; |
There was a problem hiding this comment.
Nit: no min_len on uri and no max_items on action_names (the Union backend caps it at 100). Could be enforced with buf.validate.
Cap action_names at 100 and require a uri with buf.validate, and drop the claim that the server clears fork on a non-recovery run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
The case dialled real AWS S3 for a bucket named flyte with a fake access key, and AWS now answers GetBucketLocation with 403 InvalidAccessKeyId, which fails the test on main too. Setting an endpoint and region makes stow skip the region and location lookups, so it only checks the secret key file is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
The overrides carry code as well as inputs, so drop "Input" from the names: ActionOverrides, ActionOverridesData, ActionOverrideStatus, Recover.action_overrides, RunDetails.action_override_status, and override_applied on ActionMetadata and actions.Action. Field numbers are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
|
I think this can shrink to the overrides file format plus the display fields. Two existing mechanisms already cover the rest:
So the RunSpec needs nothing new for overrides. Remove
Keep
Separately, the |
Per review (#8234 (comment)): the overrides file's URI travels in the root action's Inputs.context under "_u_action_overrides", which reaches every action, including ones on reusable containers, and the client lists the actions that must re-run in the existing Recover.force_rerun_actions. So the RunSpec needs nothing new: - Remove Recover.action_overrides and ActionOverrides, Recover.implied_rerun_actions, RunDetails.action_override_status and ActionOverrideStatus, and CreateRunResponse.warnings. - Add ActionOverridesData.ancestor_actions, which a parent enqueues without a cache key. - Reword the override_applied and ActionOverridesData comments. - Drop the stow_store_test.go change: main fixed it in #8374. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
Per review (#8234 (comment)): the overrides file's URI travels in the root action's Inputs.context under "_u_action_overrides", which reaches every action, including ones on reusable containers, and the client lists the actions that must re-run in the existing Recover.force_rerun_actions. So the RunSpec needs nothing new: - Remove Recover.action_overrides and ActionOverrides, Recover.implied_rerun_actions, RunDetails.action_override_status and ActionOverrideStatus, and CreateRunResponse.warnings. - Add ActionOverridesData.ancestor_actions, which a parent enqueues without a cache key. - Reword the override_applied and ActionOverridesData comments. - Drop the stow_store_test.go change: main fixed it in #8374. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> (cherry picked from commit 8815e81)
|
Thanks @kumare3, addressed in 8815e81 (review):
Verified end to end on a devbox (nested overrides under cached ancestors, a reusable-container parent, rerun/fork of a fork, code-only fork of a cached a0); see Testing in the description. |
| repeated string force_rerun_actions = 1 [(buf.validate.field).repeated.items.string.min_len = 1]; | ||
|
|
||
| // Field numbers of the removed action_overrides and implied_rerun_actions. | ||
| reserved 2, 3; |
There was a problem hiding this comment.
we dont need to reserve. this was never merged
| // action, and runs that action from its replacement code bundle, if any. The | ||
| // client also lists every overridden action and its ancestors in | ||
| // Recover.force_rerun_actions, so none of them is recovered. | ||
| message ActionOverridesData { |
There was a problem hiding this comment.
It comes from the client that creates the fork. Today that's the Union console's Fork form; the SDK can do the same through custom_context. The control plane never parses it.
Write (fork client, at CreateRun):
-
It uploads each edited action's partial
Inputs(and any replacement code bundle.tar.gz) to the data plane with the regular upload path. -
It builds an
ActionOverridesData(input_uris,code_bundles,ancestor_actions), serializes it, and uploads it as one more file. -
It sends
CreateRunwith aRECOVERrelation:- the file's URI goes on the root
Inputs.contextunder_u_action_overrides - the overridden actions and their ancestors go in
RunSpec.recover.force_rerun_actions
Console code:
clientsv2/src/components/LaunchForm/fork/useForkSubmit.tsandbuildLaunchRunSubmitPayload.tsin unionai/cloud#19498. - the file's URI goes on the root
Carry (backend, unchanged): Inputs.context is already inherited by every action a parent spawns, so the URI reaches each action of the run with no new plumbing. The backend only uses force_rerun_actions, to skip recovery for those actions. It never opens the overrides file.
Read (SDK remote controller, in the parent's pod): when a parent submits a sub-action, it reads _u_action_overrides from its task context. If the child's name is in the file, it loads the override inputs and overlays them by name on the inputs it computed, and/or swaps in the replacement code bundle. Then it uploads inputs and enqueues as usual. See _action_overrides / _input_overrides.py in flyteorg/flyte-sdk#1708.
The data stays in the data plane like any other input_uri; the control plane only sees a string in the context.
- Drop the hand-rolled literal_matches_type check: the fork client converts override values through the TypeEngine against the action's interface. Keep the cheap 'input no longer exists' check. - Replace ActionOverrides with one child_overrides() helper returning the inputs, replacement code bundle, skip-cache and applied flags, shared by both submit paths of the Python controller and by the Rust controller. The Rust controller enqueues through the queue service, which cannot record override_applied, so it fails an applied override with a clear error (as the Python queue path does) and skips the cache for ancestors. - Cache parsed overrides files with functools.lru_cache keyed by URI. - Read per-action inputs with io.load_inputs and drop the retry loop (storage already retries); wrap failures in one system error. - Extract convert.overlay_named_literals, used by trigger inputs, rerun inputs and overrides. - Extract convert.prepare_new_run_inputs for building a new run's root inputs from a source run's: it carries only the new run's own overrides key and reads inputs held by URI back when needed. A rerun now always drops the key; the custom_context route is gone. flyteplugins-union's fork calls this helper instead of copying it. - Move ACTION_OVERRIDES_CONTEXT_KEY next to RESERVED_CONTEXT_KEYS and replace PROPAGATED_INTERNAL_CONTEXT_KEYS with one HIDDEN_CONTEXT_KEYS. - flyteidl2 2.0.55 (merged from main) carries flyteorg/flyte#8234, so remove the compat scaffolding: the conditional override_applied kwarg, the hasattr/cast/getattr probes and the test skips. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com>
…1708) ## What The remote controller applies a forked run's **action overrides**: new inputs and/or a replacement code bundle for specific task actions. Depends on the IDL in flyteorg/flyte#8234. Per [review](#1708 (comment)), the overrides file's location travels in `Inputs.context`, not a pod-level env var, so it works for reusable containers, where one pod serves actions from many runs. ### Where the overrides come from - A fork's client puts the URI of a serialized `ActionOverridesData` on the root action's inputs under the context key `_u_action_overrides` (`flyte._constants.ACTION_OVERRIDES_CONTEXT_KEY`). The client sets it with `flyte._internal.runtime.convert.set_action_overrides_uri`, or with `with_runcontext(custom_context=...)` on the SDK's own run path. - The key isn't in `RESERVED_CONTEXT_KEYS`, so it stays in the task context and in every child's `Inputs.context`. It's hidden from `get_custom_context()`. `flyte.custom_context(...)` and `--custom-context` refuse to set it, and a `with flyte.custom_context(...)` block can't drop it. - The controller reads the key from each call's task context and loads the file through a small cache keyed by URI. Two runs served by one process therefore apply their own overrides. - A rerun or recover of a fork drops the key, unless the caller sets one itself to create a new fork. ### Input overrides - For a child whose computed name is in `input_uris`, the controller reads that child's override `Inputs` when it enqueues the child, and overlays them by input name. - The child keeps the name computed from its **original** inputs, so it matches the source run's action, which the client listed in `force_rerun_actions`. The cache key uses the **overridden** inputs. - The enqueued `Action` sets `override_applied`. - Override values are checked against the child's interface: an input the task no longer has, or a value of the wrong kind, fails the child with a clear error. - Reading the overrides file or an action's override inputs retries transient storage errors. If it still fails, the parent fails rather than silently running with the original inputs. ### Ancestors - A child listed in `ancestor_actions` is enqueued with no cache key. Its inputs are unchanged, so a cache hit would skip the subtree that holds the override, and a write would store the new result under the old key. ### Code overrides (`code_bundles`) - For a child with a code bundle override, the controller keeps the name computed from the parent's own copy of the code, then serializes the child against the replacement bundle. Only that action, and the actions it launches, run the patched code. - The bundle is the only thing replaced. The child's interface, image, resources, cache settings and retries still come from the parent's copy of the task. - The action skips the cache, because its cache version comes from the parent's copy of the function. - An override on a deployed (reference) task, or from a parent without a source bundle, fails with a user error. Runs without the key are unaffected: no file is loaded. ## Testing - Unit tests: `tests/internal/controllers` and the runtime/context tests pass (345 passed), including the URI-keyed overrides cache, the context key's propagation and hiding, ancestors enqueued without a cache key, and an image whose flyteidl2 predates `ActionOverridesData` failing with a non-recoverable error instead of retrying. - End-to-end on a local Union devbox (backend and console from unionai/cloud#19498, task images with the flyte-sdk#1708 and flyteidl2 wheels, the fork client from flyteplugins-union#204). Workflow: `root → p1 → p2 → leaf`, every task `cache="auto"`, with a sibling at each level. - **Override 3 levels deep, all ancestors cached**: fork overriding `leaf` x=3→7 gives the expected root output. `leaf` has `override_applied`, the siblings are recovered from the source run, p1/p2/a0 re-run with the cache disabled, `force_rerun_actions` is `[leaf, p2, p1]`, and `_u_action_overrides` reaches every action. Nothing was written under the old cache keys: a fresh `root(3)` and a fresh call of p1 still hit the original results. - **Parent in a reusable container**: same fork with p2 on a reusable environment; the override is applied there and the totals match. - **Rerun of a fork, fork of a fork**: a plain rerun and a recover of a fork carry no overrides key (`fork=False`) and use the original inputs; a fork of a fork writes a new overrides file, applies its own override, and recovers the first fork's overridden leaf. - **Code-only fork of a cacheable a0**: a0 runs the new code instead of hitting the cache, its children are recovered, and a fresh run of the old code still hits the old cached result. ## Follow-up Bump the `flyteidl2` pin once flyteorg/flyte#8234 is released. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
What
IDL for forking a run: a recovery run (
RELATION_TYPE_RECOVER) that reuses the source run's succeeded actions but changes the inputs and/or the code of specific task actions, or the code of the whole run.Per review, the
RunSpeccarries nothing new for overrides. Two existing mechanisms cover it:Recover.force_rerun_actions.Inputs.contextunder the reserved key_u_action_overrides. Context reaches every action in the run, including actions served by reusable containers, where a run-level env var does not.New fields:
task.ActionOverridesData: the format of the overrides file the client writes and the SDK reads.input_uris: action name to the data-plane URI of a serializedInputsholding the inputs that change. A parent's controller reads only the override inputs of the actions it launches.code_bundles: action name to anActionCodeBundle(archive URI and version). That one action, and the actions it launches, run from this bundle instead of the one their parent passes down.ancestor_actions: the ancestors of the overridden actions, excludinga0. A parent enqueues them without a cache key, because their inputs are unchanged and a cache hit would skip the subtree that holds the override.task.Recover.fork: marks a recovery run as a fork, so a fork that only changes code can be told apart from a plain recover. Informational; it doesn't change recovery.actions.Action.override_appliedandworkflow.ActionMetadata.override_applied: set by the parent's controller when it applied an override, for display.All fields are new and optional, so existing clients and servers are unaffected.
Generated code
Regenerated Go, Python, TypeScript and Rust bindings with
buf1.58.0.Testing
make buf; regenerating on the cloud pin branch gives no diff.root → p1 → p2 → leaf, every taskcache="auto", with a sibling at each level.leafx=3→7 gives the expected root output.leafhasoverride_applied, the siblings are recovered from the source run, p1/p2/a0 re-run with the cache disabled,force_rerun_actionsis[leaf, p2, p1], and_u_action_overridesreaches every action. Nothing was written under the old cache keys: a freshroot(3)and a fresh call of p1 still hit the original results.fork=False) and use the original inputs; a fork of a fork writes a new overrides file, applies its own override, and recovers the first fork's overridden leaf.Companion PRs: flyteorg/flyte-sdk#1708 (the controller applies the overrides), unionai/flyteplugins-union#204 (the
forkclient) and unionai/cloud#19498 (backend and console).🤖 Generated with Claude Code