Repository navigation
Conversation
…ateRun Abort of a finished run (F1): AbortRun and AbortAction updated the phase to ABORTED unconditionally, so an abort that arrived after a run SUCCEEDED (e.g. racing its completion) turned it into ABORTED and the run's outputs were no longer returned. They now only abort actions that are not in a final phase (SUCCEEDED, ABORTED, RECOVERED) and return interfaces.ErrActionFinished otherwise; the service treats that as a successful no-op and pushes nothing to the abort reconciler. FAILED and TIMED_OUT stay abortable, since a retry can still move an action on from them. Duplicate CreateRun (F2): The run insert is idempotent and returns the existing run, but CreateRun still called Enqueue with the new request's inputs. Once the run's TaskAction was gone (aborted, or garbage-collected after it finished), that created a new TaskAction and executed the run again, replacing its outputs while the run kept reporting its old phase. CreateRun now returns AlreadyExists for an existing run before writing inputs or enqueueing, which is what the SDK (RunAlreadyExistsError) and the trigger executor (skip already-fired runs) already expect. GetRun now wraps sql.ErrNoRows so callers can tell "not found" from other errors. Found with Antithesis; see antithesis/FINDINGS.md on the antithesis-workload branch for the reproductions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first version left FAILED and TIMED_OUT abortable, on the assumption that a retry could move an action on from them. The executor never reports either while retries remain: an in-place restart and a restart after a max-runtime timeout both report QUEUED, so FAILED and TIMED_OUT are only reported once an action has ended. An Antithesis run of the first version caught a run that FAILED after exhausting its retries being rewritten to ABORTED by a late abort, losing its error. Abort is now a no-op for SUCCEEDED, FAILED, ABORTED, TIMED_OUT and RECOVERED. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Flyte under Deterministic Simulation Testing in Antithesis
Both bugs below were found, reproduced and triaged in Antithesis, with a workload built using the Antithesis skills. The workload drives the Flyte 2 run service through the real
flyteSDK under fault injection, and checks each run against a model.Tracking issue
Related to
#8400(an ABORTED event shadows a SUCCEEDED one) and#7872(CreateRun returns success when the enqueue didn't do what was asked). Issues for the two bugs below will be linked here.Why are the changes needed?
Two bugs in the Flyte 2 run service.
1. Aborting a finished run rewrites its outcome.
AbortRunandAbortActionset the phase toABORTEDwithout checking whether the run had already ended.SUCCEEDEDrun intoABORTED, andRun.outputs()stops returning its result;FAILEDrun intoABORTED, and its error is replaced with abort info.2.
CreateRunwith an existing run name re-executes the run.CreateRunstill enqueues it with the new request's inputs.RunAlreadyExistsError) and the trigger executor both already expectAlreadyExistshere.What changes were proposed in this pull request?
AbortRunandAbortActiononly update actions that haven't ended.SUCCEEDED,FAILED,ABORTED,TIMED_OUTorRECOVERED, they return a newinterfaces.ErrActionFinished. The service treats that as a successful no-op and pushes nothing to the abort reconciler.FAILEDandTIMED_OUTcount as final because the executor reports a retrying attempt asQUEUED.CreateRunreturnsAlreadyExistsbefore writing inputs or enqueueing.GetRunnow wrapssql.ErrNoRows, so "not found" can be told apart from other errors.Before / after
Aborting a run
ABORTEDABORTED(unchanged)SUCCEEDEDABORTED, outputs no longer returnedSUCCEEDEDwith its outputs, abort returns OKFAILED(retries exhausted)ABORTED, error replacedFAILEDwith its error, abort returns OKCreateRunwith a name that already exists (different inputs)AlreadyExistsAlreadyExists, result unchangedABORTEDAlreadyExistsBehavior change for clients: re-sending
CreateRunfor an existing name now fails withAlreadyExistsinstead of returning OK. If returning the existing run is preferred, the same check can return it with OK instead, still without enqueueing it. Happy to switch.Affected versions
Both bugs are in every Flyte 2 release since the backend cutover (#6583, first in
v2.0.14) up to and includingv2.0.56, and on currentmain(2252ea7f6). Onmain,AbortRun/AbortActionstill update with no phase guard (runs/repository/impl/action.go:331), andCreateRunstill enqueues without checking whether the run exists (runs/service/run_service.go:~405).Reproduction
Each bug reduces to a few SDK calls on a flyte-binary deployment:
Bug 1: abort rewrites a finished run
flyte.with_runcontext(mode="remote", name="f1-a").run(square, x=5), and wait untilremote.Run.get(name="f1-a").phaseisSUCCEEDED.outputs()returns 25.remote.Run.get(name="f1-a").abort(). The call returns OK.remote.Run.get(name="f1-a").phaseis nowABORTED, andoutputs()no longer returns 25. In Postgres theactionsrow hasphase = 7andabort_reasonset.FAILEDrun (boom): it becomesABORTEDand its error is replaced with abort info.Bug 2: a duplicate
CreateRunre-executes the run. The quickest path uses an aborted run, because aborting deletes the TaskAction right away:flyte.with_runcontext(mode="remote", name="f2-a").run(square, x=5), thenremote.Run.get(name="f2-a").abort().flyte.with_runcontext(mode="remote", name="f2-a").run(square, x=6)returns OK. The SDK has aRunAlreadyExistsErrorfor this case, but the server never returnsALREADY_EXISTS.f2-a-a0and its pod are created, andsquare(6)runs to completion, whileremote.Run.get(name="f2-a").phasestaysABORTED.action_eventsfor the run shows... → ABORTED → WAITING → QUEUED → RUNNING → SUCCEEDED.The same thing happens to a finished run once its TaskAction is garbage-collected (
executor.gc.maxTTL, default 1h). Our repro lowered it to 2m:With this PR, step 2 of both F2 cases fails with
RunAlreadyExistsError, and the late aborts in F1 return OK and leave the runSUCCEEDED/FAILEDwith its outputs or error.Failing assertions (Antithesis)
The workload drives the run service through the
flyteSDK, keeps a model of every run (its expected result, and whether and when it was aborted), and checks these properties against what the API returns. Two 15-minute runs on the unpatched base (eb77a26d8, v2.0.55 + 5 commits), then one with this PR, all with the same workload and properties:The "correct result" failures we traced came from the two bugs: an abort that made a SUCCEEDED run's outputs unreadable, or a duplicate that replaced the result. None of the three properties failed with the fix. Example traces, using timestamps in the run's virtual time:
wb592r3: its pod logsFlyte runtime completedat 99.95. An abort arrives at 104.35 (Received AbortRun request), and at 104.42 the server logsAborted run. The run now readsABORTED.w24b7r6, aflakytask), attempt 3 completed at 110.56 and the abort landed at 112.70. At 113.42 the SDK returned no outputs for the run, so the completed result was lost.wb592r20:square(148)succeeded with result 21904 at 111.86, and its TaskAction was later collected.CreateRunwithx=149gotSuccessfully enqueued root action. A new TaskAction and pod ran and wrote the same.../wb592r20/a0/1/outputs.pb.SUCCEEDED, 1 attempt, result 22201 (149²).Confirmed vs. false positives:
SUCCEEDED → RUNNINGwalk-back turned out to be our driver recording an older, out-of-order reply. The multiverse debugger showed the Postgres row stayedSUCCEEDED. We fixed the driver, and it didn't recur.ALREADY_EXISTSonCreateRuntoRunAlreadyExistsError("A run with the name '…' already exists"). The trigger executor (runs/scheduler/executor/trigger_executor.go:99) also expectsAlreadyExistsfor duplicates. Flyte 1 rejected duplicate execution names.How was this patch tested?
go test ./runs/...passes.GetRunwrapssql.ErrNoRows.CreateRunfor an existing run returnsAlreadyExistsand writes no inputs, creates no action and does not enqueue;AbortRunandAbortActionreturn OK onErrActionFinished.CreateRunis rejected both before and after the original's TaskAction is collected, and the result stays 25.abort()leaves a SUCCEEDED run SUCCEEDED with its outputs, and a FAILED run FAILED with its error.SUCCEEDED. Antithesis then caught aFAILEDrun being rewritten, which led to including every terminal phase.Labels
Check all the applicable boxes