Skip to content

fix(runs): don't rewrite finished runs on abort; reject CreateRun for an existing run - #8416

Draft
ggiorgis1 wants to merge 2 commits into
flyteorg:mainfrom
ggiorgis1:fix/abort-terminal-and-duplicate-run
Draft

ggiorgis1 wants to merge 2 commits into
flyteorg:mainfrom
ggiorgis1:fix/abort-terminal-and-duplicate-run

Conversation

@ggiorgis1

@ggiorgis1 ggiorgis1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Flyte under Deterministic Simulation Testing in Antithesis

Both bugs below were found, reproduced and triaged in Antithesis, with a workload built using the Antithesis skills. The workload drives the Flyte 2 run service through the real flyte SDK under fault injection, and checks each run against a model.

Tracking issue

Related to #8400 (an ABORTED event shadows a SUCCEEDED one) and #7872 (CreateRun returns success when the enqueue didn't do what was asked). Issues for the two bugs below will be linked here.

Why are the changes needed?

Two bugs in the Flyte 2 run service.

1. Aborting a finished run rewrites its outcome.

  • AbortRun and AbortAction set the phase to ABORTED without checking whether the run had already ended.
  • An abort that lands after the run finished, typically one that races its completion, turns:
    • a SUCCEEDED run into ABORTED, and Run.outputs() stops returning its result;
    • a FAILED run into ABORTED, and its error is replaced with abort info.

2. CreateRun with an existing run name re-executes the run.

  • For an existing name, the insert returns the existing run, but CreateRun still enqueues it with the new request's inputs.
  • Once the run's TaskAction is gone (aborted, or garbage-collected about 1h after it finished), the task runs again. Its outputs replace the original result, and the run still reports its old phase with 1 attempt.
  • The SDK (RunAlreadyExistsError) and the trigger executor both already expect AlreadyExists here.

What changes were proposed in this pull request?

  • Abort: AbortRun and AbortAction only update actions that haven't ended.
    • For a run or action already in SUCCEEDED, FAILED, ABORTED, TIMED_OUT or RECOVERED, they return a new interfaces.ErrActionFinished. The service treats that as a successful no-op and pushes nothing to the abort reconciler.
    • FAILED and TIMED_OUT count as final because the executor reports a retrying attempt as QUEUED.
  • CreateRun: if a run with the name already exists, CreateRun returns AlreadyExists before writing inputs or enqueueing. GetRun now wraps sql.ErrNoRows, so "not found" can be told apart from other errors.

Before / after

Aborting a run

Run state when the abort arrives Before After
Running / queued ABORTED ABORTED (unchanged)
SUCCEEDED ❌ becomes ABORTED, outputs no longer returned ✅ stays SUCCEEDED with its outputs, abort returns OK
FAILED (retries exhausted) ❌ becomes ABORTED, error replaced ✅ stays FAILED with its error, abort returns OK

CreateRun with a name that already exists (different inputs)

Existing run Before After
Finished, TaskAction still present ⚠️ OK, but nothing runs ✅ AlreadyExists
Finished, TaskAction garbage-collected ❌ OK, task re-runs with the new inputs, result replaced (25 → 36) ✅ AlreadyExists, result unchanged
Aborted ❌ OK, task runs while the run still shows ABORTED ✅ AlreadyExists
No such run Normal run Normal run (unchanged)

Behavior change for clients: re-sending CreateRun for an existing name now fails with AlreadyExists instead of returning OK. If returning the existing run is preferred, the same check can return it with OK instead, still without enqueueing it. Happy to switch.

Affected versions

Both bugs are in every Flyte 2 release since the backend cutover (#6583, first in v2.0.14) up to and including v2.0.56, and on current main (2252ea7f6). On main, AbortRun/AbortAction still update with no phase guard (runs/repository/impl/action.go:331), and CreateRun still enqueues without checking whether the run exists (runs/service/run_service.go:~405).

Reproduction

Each bug reduces to a few SDK calls on a flyte-binary deployment:

import flyte, flyte.remote as remote

env = flyte.TaskEnvironment(name="repro")

@env.task
async def square(x: int) -> int:
    return x * x

@env.task
async def boom() -> int:
    raise ValueError("fails")

flyte.init(endpoint=..., insecure=True, project="flytesnacks", domain="development")

Bug 1: abort rewrites a finished run

  1. flyte.with_runcontext(mode="remote", name="f1-a").run(square, x=5), and wait until remote.Run.get(name="f1-a").phase is SUCCEEDED. outputs() returns 25.
  2. remote.Run.get(name="f1-a").abort(). The call returns OK.
  3. remote.Run.get(name="f1-a").phase is now ABORTED, and outputs() no longer returns 25. In Postgres the actions row has phase = 7 and abort_reason set.
  4. The same happens to a FAILED run (boom): it becomes ABORTED and its error is replaced with abort info.

Bug 2: a duplicate CreateRun re-executes the run. The quickest path uses an aborted run, because aborting deletes the TaskAction right away:

  1. flyte.with_runcontext(mode="remote", name="f2-a").run(square, x=5), then remote.Run.get(name="f2-a").abort().
  2. flyte.with_runcontext(mode="remote", name="f2-a").run(square, x=6) returns OK. The SDK has a RunAlreadyExistsError for this case, but the server never returns ALREADY_EXISTS.
  3. A new TaskAction f2-a-a0 and its pod are created, and square(6) runs to completion, while remote.Run.get(name="f2-a").phase stays ABORTED. action_events for the run shows ... → ABORTED → WAITING → QUEUED → RUNNING → SUCCEEDED.

The same thing happens to a finished run once its TaskAction is garbage-collected (executor.gc.maxTTL, default 1h). Our repro lowered it to 2m:

== resubmit before the TaskAction is collected (control)
  submitted square(5)            -> SUCCEEDED, 1 attempt, result 25
  resubmitted square(6), same name: CreateRun returned OK
  20s later                      -> SUCCEEDED, 1 attempt, result 25   (OK, but nothing ran)
== resubmit after the TaskAction is collected
  submitted square(5)            -> SUCCEEDED, 1 attempt, result 25
  TaskAction collected after 204s
  resubmitted square(6), same name: CreateRun returned OK
  20s later                      -> SUCCEEDED, 1 attempt, result 36   (re-executed, result replaced)

With this PR, step 2 of both F2 cases fails with RunAlreadyExistsError, and the late aborts in F1 return OK and leave the run SUCCEEDED/FAILED with its outputs or error.

Failing assertions (Antithesis)

The workload drives the run service through the flyte SDK, keeps a model of every run (its expected result, and whether and when it was aborted), and checks these properties against what the API returns. Two 15-minute runs on the unpatched base (eb77a26d8, v2.0.55 + 5 commits), then one with this PR, all with the same workload and properties:

Property (Antithesis "always" assertion) Run 1, unpatched Run 2, unpatched With this PR
an abort never changes a finished run's outcome ❌ 7 failures / 135 checks ❌ 10 / 165 ✅ 0 / 127
a duplicate submission never changes a run's result ❌ 1 / 40 ❌ 4 / 51 ✅ 0 / 51
a SUCCEEDED run returns the correct result ❌ 5 / 130 ❌ 4 / 165 ✅ 0 / 115
a duplicate submission of an existing run is never accepted (added with the fix) (added with the fix) ✅ 0 / 81

The "correct result" failures we traced came from the two bugs: an abort that made a SUCCEEDED run's outputs unreadable, or a duplicate that replaced the result. None of the three properties failed with the fix. Example traces, using timestamps in the run's virtual time:

  • Bug 1, run wb592r3: its pod logs Flyte runtime completed at 99.95. An abort arrives at 104.35 (Received AbortRun request), and at 104.42 the server logs Aborted run. The run now reads ABORTED.
    • In another run (w24b7r6, a flaky task), attempt 3 completed at 110.56 and the abort landed at 112.70. At 113.42 the SDK returned no outputs for the run, so the completed result was lost.
  • Bug 2, run wb592r20:
    • square(148) succeeded with result 21904 at 111.86, and its TaskAction was later collected.
    • At 283.99 a duplicate CreateRun with x=149 got Successfully enqueued root action. A new TaskAction and pod ran and wrote the same .../wb592r20/a0/1/outputs.pb.
    • At 296.22 the run reads SUCCEEDED, 1 attempt, result 22201 (149²).

Confirmed vs. false positives:

  • Found and reproduced in Antithesis, then reduced to the minimal SDK steps above. Faults (pod kills, network partitions, thread pauses) made the races more frequent, but neither bug depends on them.
  • One failure was ruled out and isn't claimed here. An apparent SUCCEEDED → RUNNING walk-back turned out to be our driver recording an older, out-of-order reply. The multiverse debugger showed the Postgres row stayed SUCCEEDED. We fixed the driver, and it didn't recur.
  • Intended behavior, which is why we read these as bugs and not design choices:
    • Bug 2: the SDK maps ALREADY_EXISTS on CreateRun to RunAlreadyExistsError ("A run with the name '…' already exists"). The trigger executor (runs/scheduler/executor/trigger_executor.go:99) also expects AlreadyExists for duplicates. Flyte 1 rejected duplicate execution names.
    • Bug 1: Flyte 1's flyteadmin rejected aborts of terminated executions. Nothing in the docs or tests treats aborting a finished run as changing its outcome.
  • Not claimed here: an intermittent exit-139 crash of task pods at exit, also seen in these runs. It isn't confirmed to be a Flyte bug and is out of scope for this PR.

How was this patch tested?

  • Unit tests: the new tests fail without the fix, and go test ./runs/... passes.
    • Repository tests, against embedded Postgres: aborting a run in each terminal phase is a no-op; queued, initializing and running runs are still aborted; a missing run or action is still not found; GetRun wraps sql.ErrNoRows.
    • Service tests: CreateRun for an existing run returns AlreadyExists and writes no inputs, creates no action and does not enqueue; AbortRun and AbortAction return OK on ErrActionFinished.
  • End to end on flyte-binary (k3s) with the SDK:
    • A duplicate CreateRun is rejected both before and after the original's TaskAction is collected, and the result stays 25.
    • A late abort() leaves a SUCCEEDED run SUCCEEDED with its outputs, and a FAILED run FAILED with its error.
  • Antithesis: see "Failing assertions" above.
    • A first version of this fix covered only SUCCEEDED. Antithesis then caught a FAILED run being rewritten, which led to including every terminal phase.
    • With the final change, all of the listed properties pass, and the targeted races were all exercised: abort racing completion, duplicates of aborted runs, and duplicates of finished runs.

Labels

  • fixed

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

ggiorgis1 and others added 2 commits November 16, 2026 17:39
…ateRun

Abort of a finished run (F1):
AbortRun and AbortAction updated the phase to ABORTED unconditionally, so an
abort that arrived after a run SUCCEEDED (e.g. racing its completion) turned it
into ABORTED and the run's outputs were no longer returned. They now only abort
actions that are not in a final phase (SUCCEEDED, ABORTED, RECOVERED) and return
interfaces.ErrActionFinished otherwise; the service treats that as a successful
no-op and pushes nothing to the abort reconciler. FAILED and TIMED_OUT stay
abortable, since a retry can still move an action on from them.

Duplicate CreateRun (F2):
The run insert is idempotent and returns the existing run, but CreateRun still
called Enqueue with the new request's inputs. Once the run's TaskAction was gone
(aborted, or garbage-collected after it finished), that created a new TaskAction
and executed the run again, replacing its outputs while the run kept reporting
its old phase. CreateRun now returns AlreadyExists for an existing run before
writing inputs or enqueueing, which is what the SDK (RunAlreadyExistsError) and
the trigger executor (skip already-fired runs) already expect. GetRun now wraps
sql.ErrNoRows so callers can tell "not found" from other errors.

Found with Antithesis; see antithesis/FINDINGS.md on the antithesis-workload
branch for the reproductions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first version left FAILED and TIMED_OUT abortable, on the assumption that
a retry could move an action on from them. The executor never reports either
while retries remain: an in-place restart and a restart after a max-runtime
timeout both report QUEUED, so FAILED and TIMED_OUT are only reported once an
action has ended. An Antithesis run of the first version caught a run that
FAILED after exhausting its retries being rewritten to ABORTED by a late abort,
losing its error. Abort is now a no-op for SUCCEEDED, FAILED, ABORTED,
TIMED_OUT and RECOVERED.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ggiorgis1 ggiorgis1 changed the title Fix/abort terminal and duplicate run fix(runs): don't rewrite finished runs on abort; reject CreateRun for an existing run Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant