One stuck `running` job row permanently clogs the queue — worker trusts exit code 0 and never reaps dead/hung jobs

[kenking2536]

Summary

A single job row left at status='running' in the DB blocks the entire queue for that gpu_ids group forever: processQueue treats any running row as "busy" and continues, so no further job for those GPUs is ever dispatched. This is reproducible, has happened repeatedly, and until now the only recovery was a manual DB/row cleanup.

Root cause (two compounding gaps)

1. Non-UI trainer types never write the DB. If a job is started with a trainer type that is not the UI-managed diffusion trainer (e.g. a CLI-style trainer), the python side never writes a terminal status. The worker then trusts python to do it: in ui/cron/actions/startJob.ts the exit path is effectively

   if (code === 0) return;   // assume python already set completed

   so a clean exit (or a process that never ran the UI write path) leaves the row running forever.

2. No fallback reaper. The worker's exit listener is lost on worker restart; a process can also hang during shutdown after training finished (log goes silent, pid still alive); or the process dies (crash/OOM/reboot) without writing its own status. None of these states is ever re-examined, so the stuck row stays.

Both scenarios hit us on a 4-GPU FLUX setup: once, after a plain exit-0 run of a non-UI trainer type; and again after a worker restart. In both cases the queue for 0,1,2,3 never dispatched again until the row was marked manually.

Workaround (working, what we ship in production)

A reaper sweep at the top of processQueue (ui/cron/actions/processQueue.ts), run at most every 30 s:

const REAP_INTERVAL_MS = 30 * 1000;
const NO_PID_GRACE_MS = 5 * 60 * 1000;      // 'running' w/o pid: launch grace
const LOG_SILENCE_KILL_MS = 10 * 60 * 1000; // live pid + log silent > 10 min => hung
let lastReapAt = 0;

const isReapPidAlive = (pid: number): boolean => {
  try { process.kill(pid, 0); return true; }
  catch (e: any) { return e?.code === 'EPERM'; }
};

export async function reapZombieJobs() {
  const now = Date.now();
  if (now - lastReapAt < REAP_INTERVAL_MS) return;
  lastReapAt = now;

  const stuck: Job[] = await prisma.job.findMany({ where: { status: 'running' } });
  if (stuck.length === 0) return;

  let trainingRoot: string | null = null;
  for (const job of stuck) {
    try {
      if (job.pid != null && isReapPidAlive(job.pid)) {
        if (trainingRoot === null) trainingRoot = await getTrainingFolder();
        const logPath = path.join(trainingRoot, job.name, 'log.txt');
        let silent = false;
        try { silent = now - fs.statSync(logPath).mtimeMs > LOG_SILENCE_KILL_MS; }
        catch { silent = false; } // no log file yet => fresh launch, never kill
        if (!silent) continue;
        try { process.kill(job.pid, 9); } catch { /* already gone */ }
        await prisma.job.updateMany({
          where: { id: job.id, status: 'running' },
          data: { status: 'error', pid: null,
                  info: `[reaper] pid ${job.pid} alive but log silent >10min; killed` },
        });
        continue;
      }
      if (job.pid != null) {
        await prisma.job.updateMany({
          where: { id: job.id, status: 'running' },
          data: { status: 'error', pid: null,
                  info: `[reaper] pid ${job.pid} no longer alive; job left 'running'` },
        });
        continue;
      }
      const ageMs = now - new Date(job.updated_at).getTime();
      if (ageMs < NO_PID_GRACE_MS) continue;
      await prisma.job.updateMany({
        where: { id: job.id, status: 'running' },
        data: { status: 'error',
                info: '[reaper] running with no pid after grace period; launch likely failed' },
      });
    } catch (e) {
      console.error(`[reaper] failed to inspect job ${job.id}:`, e);
    }
  }
}

// at the top of processQueue():
try { await reapZombieJobs(); }
catch (e) { console.error('reap sweep failed:', e); }

Safety properties we consider essential (and would keep upstream):

• Every write is updateMany guarded by status: 'running', so the reaper can never clobber a terminal status the python side just wrote (race-safe).
• Live pids are only touched after a 10-minute log-silence threshold; missing log file = fresh launch = never kill.
• pid == null rows get a 5-minute grace (pid is written a few hundred ms after spawn).
• The sweep runs before queue dispatch, so a zombie can never hold gpu_ids busy for a round.

Verified in production on a 4×16 GB FLUX training box: stuck rows now flip to error within one sweep, the queue resumes, and no healthy job has ever been touched.

Suggested upstream fix

1. Don't trust exit code 0 as proof of a terminal DB state: after the child exits, the worker should verify the row is no longer running and, if it is, mark it from the exit code (0 → completed only if the trainer guarantees it wrote the row; otherwise error with the exit code recorded).
2. Add a reaper with the semantics above (pid liveness + log-silence kill + no-pid grace), idempotent and race-safe.
3. Even minimally: processQueue should time-box the "busy" check — a running row older than N minutes with a dead pid should be treated as an error, not as busy-forever.

Environment

• ai-toolkit main commit db8dbd6 (2026-09-15); first observed 0.13.5 (2026-09-06), re-verified on 0.13.6 and current main.
• Ubuntu 26.04, UI production mode (npm start: worker + fileServer), Prisma DB.
• Repro: start any non-UI-managed trainer job (e.g. CLI-style sd_trainer flow) for gpu_ids 0,1,2,3, let it exit 0 without a UI DB write → submit a normal next job → it never starts.

---