GPU fault classification: typed failure codes from Xid events

[samhita-alla]

Why

Two things stand between a GPU fault on a node and the user who has to act on it.

The first is that failure codes get dropped on the way out. flytek8s.DemystifyFailure works out a code from the pod status and the container states and then throws it away on the system path, reporting every system error as Interrupted, so a graceful node shutdown reaches the user with no trace of Shutdown or NodeShutdown. toActionErrorInfo in the executor then builds a workflow.ErrorInfo out of the message and the kind only, so even OOMKilled never makes it onto the action event.

The second is that nothing classifies GPU faults at all. Where a node-level agent records NVIDIA Xid and NVSwitch SXid faults as Warning Events on the task pod (the message format is defined in the new gpufault package), the executor already forwards pod events into the attempt's cluster events, but the failure the user sees is still whatever exit code the container produced. A GPU that fell off the bus and a program with a bug look the same. A hardware fault is the platform's problem, not the user's, and the failure should say so instead of burning a user retry.

Known limitation, tracked as a follow-up: classification covers tasks whose tracked resource is the Pod itself. CRD-backed tasks (RayJob, PyTorchJob, MPIJob) record the fault event on a worker pod while the executor tracks the CRD, so they are not classified yet; covering them needs per-plugin child-pod discovery in the event lookup, implemented in #7913 (stacked on this PR).

What

• New shared package flyteplugins/go/tasks/pluginmachinery/gpufault. It holds the event-message contract (format, parser, Xid name and severity tables) so that emitters and consumers share one definition, the conversions to and from core.GpuFault, and the classification rules. The wire format is unchanged byte for byte, because events already recorded on running clusters have to keep parsing.

• ClassifyFailure(phaseInfo, faults) folds the faults observed on a failed attempt's pod into the failure the plugin reported:

  worst severity seen	phase	error kind	code	message	gpu_fault
  CRITICAL (79, 48/94/95/140, 63/64, 74, 119/120, every SXid)	system retryable	SYSTEM	CodeFor(fault), e.g. GpuFallenOffBus	fault sentence prepended	first critical fault
  USER (13, 31, 43, 45), plugin had a specific code (OOMKilled, NodeShutdown)	unchanged	unchanged	unchanged	fault sentence prepended	first user fault
  USER (13, 31, 43, 45), plugin code was generic (UnknownError, Interrupted, Error, empty, a bare exit status)	unchanged	USER	GpuXidError	fault sentence prepended	first user fault
  WARN only	unchanged	unchanged	unchanged	unchanged	first fault

  A critical Xid means the device or the node is not trustworthy and the workload did not cause it, so it must not consume a user retry, and the reschedule wants to land elsewhere once phase 3 quarantines the node. A user Xid is the workload's own doing (NVIDIA documents 13, 31, 43 and 45 as application-caused: illegal address, out-of-bounds access, illegal instruction, a dead context preempted). The rule for it: if the plugin had a reason of its own for the failure, that verdict stands in full and the fault only adds the driver's sentence to the message. If all the plugin could say was that the pod died (Interrupted is the kubelet-left-no-reason guess, and it guesses SYSTEM), the recorded fault is the explanation, so the code becomes GpuXidError and the kind becomes USER: a kernel that faults its own GPU will fault again, and replaying it against the system retry budget burns up to thirty attempts, each breaking another CUDA context, to reach the same answer. Retryable versus permanent stays the plugin's call either way, so a permanent failure is never downgraded to retryable. The cost of this rule being wrong is benign: the rare Xid 31 that is really a bad device shows up as a user error with the exact Xid, GPU and driver sentence on it, and the user reruns; the cost of the opposite rule is the retry storm. The error URI, timestamp, worker, recoverability, task info and phase version of the original failure are preserved in every case.

• DemystifyFailure reports the code it worked out on the system path instead of flattening everything to Interrupted. The SIGKILL branch and the branch where the kubelet recorded nothing before the node went away still report Interrupted, now explicitly.

• toActionErrorInfo carries Code and GpuFault onto workflow.ErrorInfo. The CR-persisted ErrorState already round-trips the code, so that path needed no change.

• Executor wiring: PluginManager.classifyGpuFailure runs after attachRecentObjectEvents when the attempt failed and the resource is a Pod. It lists every event on the pod for the whole attempt rather than only since the event watermark, because the Xid is usually recorded rounds before the pod status catches up with it. Nothing else in the executor sets ExecutionError.gpu_fault.

Behaviour change

situation	before	after
critical Xid (79) during a task that failed	UnknownError / Interrupted, user retryable, message is the pod status	GpuFallenOffBus, system retryable, message starts with [gpu-health] [CRITICAL] Xid 79 (GPU has fallen off the bus) on GPU 0 GPU-…, gpu_fault set
Xid 31 and the container exits 1	user retryable, code UnknownError	user retryable, code GpuXidError, message starts with [gpu-health] [USER] Xid 31 (GPU memory page fault) …, gpu_fault set
Xid 31 and the pod is killed (exit 137, no error record)	system retryable, code Interrupted, retried against the system budget	user retryable, code GpuXidError, same message shape, fails once unless the task has retries
Xid 31 on a task that OOMed	user permanent, code OOMKilled	unchanged kind and code, message starts with the Xid sentence, gpu_fault set
WARN Xid (92) on a task that OOMed	user permanent, code OOMKilled	unchanged, plus gpu_fault for the console
no Xid at all	unchanged	unchanged
pod failed with status reason NodeShutdown	system retryable, code Interrupted	system retryable, code NodeShutdown

Tests

• go build ./..., go vet ./flyteplugins/... ./executor/...
• go test ./flyteplugins/go/tasks/pluginmachinery/... ./flyteplugins/go/tasks/plugins/k8s/pod/... ./executor/pkg/...
• golangci-lint run --tests=false (v2.12.2, the pinned version) in flyteplugins/ and executor/ on the touched packages: no new findings.
• New table tests: message round trip and Xid tables, proto conversions, every classification rule, DemystifyFailure system codes, toActionErrorInfo, and classifyGpuFailure against a fake event watcher.

Stack: #7878 (base) → this.

🤖 Generated with Claude Code

• main
  • GPU fault classification: typed failure codes from Xid events 👈
    • GPU fault classification: cover the child pods of distributed tasks #7913

[samhita-alla]

Rebased onto the updated base (#7878 rebased onto main after #7882; gpu_fault is now ClusterEvent field 7). The new TestAddObjectMetadata_ManagedLabel from main is preserved alongside the classification tests.

[pingsutw]

should we do?

firstSeen := event.CreatedAt                                                                                                                                                                                        
if firstSeen.IsZero() { firstSeen = event.LastObservedAt }
if lead := failureAt.Sub(firstSeen); lead > window || lead < -slack { skip }

because Fault starts 1 min before the container dies, keeps
firing for 11 min after → lead ≈ -11m < -2m → dropped. The louder and longer the hardware screams, the more certain the fault is discarded.

[samhita-alla]

now i'm treating the event as active over [CreatedAt, LastObservedAt] and keeping it when that interval overlaps [failureAt - 30m, failureAt + 2m] instead of just using CreatedAt because a fault first recorded 40 minutes before the failure (outside the 30-minute window) can still be firing at failure time.

[pingsutw]

The pod plugin stamps it from the // container's termination

Should we check if phaseInfo is terminated or not? otherwise, the failureAt may become container start time

[samhita-alla]

terminal check is already there. classifyGpuFailure returns early unless phaseInfo.Phase().IsFailure(), so we never get here on a non-terminal phase. but OccurredAt comes from GetLastTransitionOccurredAt which returns a running container's StartedAt when nothing terminated, so a pod evicted while running anchored the window on a start time hours in the past and discarded every real fault.

the anchor is now derived from the pod itself: latest terminated main-container FinishedAt, then DeletionTimestamp.