Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions flyteidl2/actions/actions_service.proto
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,10 @@ message Action {
// identifies the source action (source run id + action name) whose result was reused.
// never set by clients on enqueue.
flyteidl2.common.ActionIdentifier recovered_from = 10;

// set by the parent's controller when this action's inputs or code were overridden
// from the run's overrides file (see flyteidl2.task.ActionOverridesData).
bool override_applied = 11;
}

// ============================================================================
Expand Down
39 changes: 39 additions & 0 deletions flyteidl2/task/common.proto
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,45 @@ message Inputs {
repeated flyteidl2.core.KeyValuePair context = 2;
}

// ActionOverridesData is the file a fork's client writes to the data plane to
// override the inputs or code of task actions of the run being recovered. Its URI
// travels in the root action's Inputs.context under the reserved key
// "_u_action_overrides", which every action of the run inherits. The controller
// of each overridden action's parent reads that action's override inputs from
// their path, overlays them, by input name, on the inputs it computes for that
// action, and runs that action from its replacement code bundle, if any. The
// client also lists every overridden action and its ancestors in
// Recover.force_rerun_actions, so none of them is recovered.
message ActionOverridesData {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this passed from

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It comes from the client that creates the fork. Today that's the Union console's Fork form; the SDK can do the same through custom_context. The control plane never parses it.

Write (fork client, at CreateRun):

  1. It uploads each edited action's partial Inputs (and any replacement code bundle .tar.gz) to the data plane with the regular upload path.

  2. It builds an ActionOverridesData (input_uris, code_bundles, ancestor_actions), serializes it, and uploads it as one more file.

  3. It sends CreateRun with a RECOVER relation:

    • the file's URI goes on the root Inputs.context under _u_action_overrides
    • the overridden actions and their ancestors go in RunSpec.recover.force_rerun_actions

    Console code: clientsv2/src/components/LaunchForm/fork/useForkSubmit.ts and buildLaunchRunSubmitPayload.ts in unionai/cloud#19498.

Carry (backend, unchanged): Inputs.context is already inherited by every action a parent spawns, so the URI reaches each action of the run with no new plumbing. The backend only uses force_rerun_actions, to skip recovery for those actions. It never opens the overrides file.

Read (SDK remote controller, in the parent's pod): when a parent submits a sub-action, it reads _u_action_overrides from its task context. If the child's name is in the file, it loads the override inputs and overlays them by name on the inputs it computed, and/or swaps in the replacement code bundle. Then it uploads inputs and enqueues as usual. See _action_overrides / _input_overrides.py in flyteorg/flyte-sdk#1708.

The data stays in the data plane like any other input_uri; the control plane only sees a string in the context.

// Action name in the run being recovered -> URI of a serialized Inputs in the
// data plane holding the inputs that change (partial inputs). Like an action's
// input_uri, the values stay in the data plane and are only read by the parent
// that launches the action.
map<string, string> input_uris = 1;

// Action name in the run being recovered -> the code bundle that action runs
// instead of the one its parent passes down. Only that action, and the actions
// it launches, run the replacement code; the rest of the run keeps its own.
map<string, ActionCodeBundle> code_bundles = 2;

// Ancestors of the overridden actions, up to (excluding) the root action a0.
// Their inputs are unchanged, so a parent enqueues them without a cache key:
// a cache hit would skip the subtree that holds the override, and a write
// would store the new result under the old key.
repeated string ancestor_actions = 3;
}

// ActionCodeBundle is a code bundle archive (.tar.gz) in the data plane that one
// action of a fork runs in place of its parent's.
message ActionCodeBundle {
// URI of the archive.
string tgz = 1 [(buf.validate.field).string.min_len = 1];

// Version of the archive's contents (for example its MD5 digest), recorded as
// the action's code bundle version.
string version = 2;
}

// Declares that a named output of the producing action is a new artifact
// version. The output's literal (in Outputs.literals) is the artifact's value;
// org/project/domain come from the producing action's scope at registration.
Expand Down
9 changes: 9 additions & 0 deletions flyteidl2/task/run.proto
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,15 @@ message Recover {
// subtree). A listed condition action re-pauses and waits for a new signal.
// Unknown names are ignored.
repeated string force_rerun_actions = 1 [(buf.validate.field).repeated.items.string.min_len = 1];

// Field numbers of the removed action_overrides and implied_rerun_actions.
reserved 2, 3;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we dont need to reserve. this was never merged


// Marks this recovery run as a fork: it reuses the run being recovered but runs
// changed code or inputs, rather than retrying it unchanged. Set by the client
// that creates the fork. Informational: it does not change how the run is
// recovered.
bool fork = 4;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: "clears it on a run that is not a recovery". The Union backend rejects any recover on a non-recovery run (InvalidArgument), so it never gets cleared.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4407aed: the comment no longer says the server clears it.

}

// TaskSpecSource records how a run's task spec was provided at creation time, so
Expand Down
4 changes: 4 additions & 0 deletions flyteidl2/workflow/run_definition.proto
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,10 @@ message ActionMetadata {
// Set only when this action was recovered from a prior run instead of executed
// here: the source action (source run id + action name) whose result was reused.
common.ActionIdentifier recovered_from = 18;

// Set when this action was enqueued with inputs or code overridden from the
// run's overrides file (see flyteidl2.task.ActionOverridesData).
bool override_applied = 19;
}

enum ActionType {
Expand Down
270 changes: 141 additions & 129 deletions gen/go/flyteidl2/actions/actions_service.pb.go

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions gen/go/flyteidl2/actions/actions_service.pb.validate.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

409 changes: 306 additions & 103 deletions gen/go/flyteidl2/task/common.pb.go

Large diffs are not rendered by default.

254 changes: 254 additions & 0 deletions gen/go/flyteidl2/task/common.pb.validate.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 16 additions & 2 deletions gen/go/flyteidl2/task/run.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading