[ogabrielluiz]

Adds the opt-in scaled backend so background flows run off the API workers, decoupled and distributed across replicas. Same BackgroundExecutionService facade, the endpoint never knows which backend is active. The queue is the database itself: no broker, no redis, so deployments need nothing beyond the DB the API already uses.

This sits directly on the release branch. The facade refactor that makes the default in-process path a first-class backend (#15018) is stacked on top of it rather than underneath.

What it adds

• A langflow worker CLI process that lease-claims QUEUED job rows off the shared database (claim_next_queued_lease, the same exact-heartbeat conditional UPDATE every other single-flight guard uses), runs the same runner the default backend uses, and persists durable milestones to job_events so any API replica can reattach to any job.
• A lease + heartbeat + periodic watchdog: a worker stamps a heartbeat while it runs, and a dead worker's in-flight job is reconciled by the watchdog without needing a restart. Reconciliation is liveness-aware, so a booting or scaled-up worker never re-claims or fails a job another worker is mid-running, and the worker_lost transition is a conditional UPDATE so concurrent watchdogs cannot write duplicate terminal events.
• Cross-replica reattach: replay the durable job_events from the DB, then poll the same table past the replayed boundary (no gap, no double delivery, Last-Event-ID contract preserved). A grace pass on terminal status covers the paths that flip the status before appending their terminal milestone. Tails end on SUSPENDED too, so a HITL pause never strands a reader.
• Stop is the durable execution_signals STOP row that the worker polls at vertex boundaries.
• A composite (status, type, created_timestamp) index backing the claim poll and watchdog scan. ix_job_status_created, which shipped with the job metrics, cannot serve either query because the type filter sits between its two sorted columns.
• Docs for running the fleet: a "Scale background runs" section, a background settings reference table (including the job retention window, which stays off until an operator sets one), and the troubleshooting entries for a multi-worker deployment.

Because the queue and the system of record are one table, the broker failure modes are structurally impossible: there is no pending/processing list to drift from the DB (a QUEUED row is enqueued by definition, so stranded-queue recovery has no equivalent) and no token-exchange protocol (the conditional UPDATE is the single flight).

Selection is the explicit LANGFLOW_BACKGROUND_BACKEND=scaled setting, decoupled from job_queue_type (redis for v1 build events no longer implies anything about the background backend). The facade logs once at wiring time in scaled mode so a fleet with zero workers is diagnosable. Scaled + SQLite works on a single host; use Postgres for multi-host fleets.

Semantics

Same guarantees as the default backend, now across processes: at-least-once for not-yet-started work, at-most-once for in-flight work by default (a crashed worker's job becomes FAILED worker_lost), and opt-in bounded retry per flow with atomic attempt accounting so concurrent watchdogs can't push a job past max_attempts.

Tradeoffs made deliberately: claim latency is bounded by the worker poll window (--idle-block-ms, default 1s) and reattach milestone latency by background_poll_interval_s (default 0.5s) instead of a broker's push; inline request globals are dropped on every scaled run (the worker hydrates from the redacted persisted row, so background runs need stored global variables).

What's proven

The former real-redis test fleet now runs against the ordinary test DB (no external service), and the real-service tier needs only Postgres:

• A real langflow worker OS subprocess claims, builds, and completes a real graph off a shared Postgres while the API side reattaches; a kill -9 mid-job is reconciled by the watchdog (fresh-lease guard proven first); a durable stop reaches the separate worker and lands CANCELLED with the signal consumed.
• Concurrency proofs: racing claimers get distinct jobs, racing watchdogs bump retry attempts exactly once and write exactly one terminal run_failed event, racing lease-claims on one row produce exactly one winner.
• 249 background-execution tests green on real Postgres with zero skips; migration consistency green on SQLite and Postgres with a single alembic head.

Three review rounds (specialist + adversarial validation per round) ran on this rework; all findings are fixed in-branch. The redis-era Locust numbers were redone against this backend on real Postgres: the claim path holds single-flight at every fleet size tested (1 to 16 workers), the composite claim-scan index is load-bearing (claim latency degrades 48x without it), and event tails stay lossless through 1000 concurrent jobs. Head-to-head under saturation the scaled backend sustains 14 to 17 completions/s at 50 to 200 concurrent users with zero lost jobs, while the in-process default collapses to 1.4/s with 35% of submissions never completing (about 10x). Known ceiling at this stage: FIFO candidate contention rises with worker count (90.9% lost-claim rate at 16 workers with candidates=5). SKIP LOCKED is the marked upgrade path, and the rest of the P0 hardening list is worker-loop claim resilience plus the worker-fleet gauges. Retention/purge and the queue metrics that were on that list have since shipped on the release branch (#15316, #15314), so this branch carries the claim-scan index against a table that retention now bounds.

Summary by CodeRabbit

• New Features

  • Added a database-backed scaled background execution mode.
  • Added a worker command for processing queued jobs in separate processes.
  • Added durable job events, cancellation signals, lease recovery, and watchdog handling.
  • Added explicit settings for backend selection and polling intervals.
  • Added load-testing support for background workflows.

• Bug Fixes

  • Improved recovery of jobs interrupted by worker failures.
  • Ensured cancellation requests are persisted and honored reliably.

• Tests

  • Added comprehensive coverage for worker execution, recovery, cancellation, concurrency, and end-to-end workflows.

[coderabbitai]

Important

Review skipped

The saved review history does not include the base for the last reviewed commit. This saved history cannot establish the base for an incremental review. Comment @coderabbitai full review to establish a new review baseline. No full review was started, and the last reviewed checkpoint was preserved.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

Walkthrough

The PR replaces Redis-based scaled background execution with a PostgreSQL-backed queue. It adds explicit backend settings, lease-based workers, durable event streaming, watchdog recovery, a worker CLI command, migrations, load testing, and real-service validation.

Changes

Scaled background execution

Layer / File(s)	Summary
Runtime and job contracts src/lfx/src/lfx/services/settings/groups/runtime.py, src/backend/base/langflow/services/jobs/service.py, src/backend/base/langflow/services/database/models/jobs/model.py, src/backend/base/langflow/alembic/versions/...	Scaled execution is selected explicitly. Job claiming, lease recovery, atomic failure, and the claim-scan index are added.
Database backend and facade wiring src/backend/base/langflow/services/background_execution/db_backend.py, factory.py, service.py, live_bus.py, src/backend/tests/unit/services/background_execution/test_db_backend.py, test_lease_watchdog.py, test_scaled_end_to_end.py, test_scaled_stop.py	DBBackgroundQueue persists stop signals, claims jobs, reconciles stale leases, and streams durable events. The service and factory use it for scaled execution.
Worker process execution src/backend/base/langflow/__main__.py, src/backend/base/langflow/services/background_execution/worker.py, src/backend/tests/unit/services/background_execution/test_worker_*.py, _subprocess_harness.py	The langflow worker command runs database-backed jobs with lease heartbeats, watchdog reconciliation, signal handling, request hydration, and teardown.
Operational validation and load testing .github/workflows/migration-validation.yml, src/backend/tests/locust/v2_background_locustfile.py, src/backend/tests/unit/background_execution/*	CI and real-service tests use PostgreSQL without Redis. Side-effect safety, subprocess recovery, and background workflow load testing are added.

Estimated code review effort: 4 (Complex) | ~60 minutes

Suggested labels: refactor

Suggested reviewers: erichare

Sequence Diagram(s)

sequenceDiagram
  participant API as BackgroundExecutionService
  participant DB as PostgreSQL
  participant Worker as langflow worker
  participant Runner as WorkerJobRunner
  API->>DB: Store queued job
  Worker->>DB: Claim lease
  Worker->>Runner: Hydrate and execute job
  Runner->>DB: Store status and events
  API->>DB: Replay durable events

Loading

Pre-merge checks: 6 · 3

Failed checks (3 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 35.59% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Quality And Coverage		Core queue, runner, stop, lease, replay, and API success/error paths are tested, but periodic watchdog and CLI signal paths lack tests; retry safety runs the graph directly, not through the worker.	Add tests for run_worker_loop with watchdog_interval_s, CLI backend and signal handling, migration index creation, and retry-safe side effects through actual claim and worker execution.
Test File Naming And Structure		The subprocess and end-to-end tests are under tests/unit; the project marks that path as unit, and no changed file uses pytest.mark.integration.	Move true integration/E2E tests to tests/integration or add the integration marker under an agreed directory; add try/finally cleanup for created worker tasks.

Passed checks (6 passed)

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Test Coverage For New Implementations		Added focused DB queue, lease watchdog, worker, factory, stop, settings, and facade tests, plus scaled end-to-end and real langflow worker subprocess tests; files use test_*.py.
Excessive Mock Usage Warning		Changed background-execution tests use no Mock or patch APIs; real database fixtures and worker subprocesses cover core behavior, with only small contract-focused fakes and environment monkeypatching.
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main changes: a database-backed scaled background execution backend and the new langflow worker command.

✨ Finishing Touches: 💡 1

📝 Generate docstrings: 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/v2-workflows-bg-scaled

_{Comment @coderabbitai help to get the list of available commands.}

[ogabrielluiz]

Stacked PR 2 of 2, based on #13507 (durable default backend). Review #13507 first. Combined view: #13505.

[github-actions]

✅ Migration Validation Passed

All migrations follow the Expand-Contract pattern correctly.

[Jkavia]

Looked at the scaled backend against the usual db-as-a-queue failure modes. Core lease/claim/watchdog looks solid. A few things:

Heartbeat and terminal write. The heartbeat is an asyncio task on the same loop as the run (_beat in runner.py), so a component that blocks the loop stops the beat and the lease goes stale while the job is still running. Another worker's watchdog then marks the job failed, with reason worker_lost. The success path writes COMPLETED unconditionally, update_job_status -> update(Job).where(Job.job_id == job_id) at service.py:1557, with no status or owner guard, while fail_in_progress_job is guarded on status=IN_PROGRESS. So the blocked run finishes and flips the row from FAILED back to COMPLETED, leaving a completed row that still carries a run_failed event, and any client that reattached in between was already told it failed. Guarding the success write on status=IN_PROGRESS AND owner=me would close it.

Clock skew. is_lease_stale compares now() on the reconciler against the worker's heartbeat_at. Across multiple hosts that depends on NTP with no skew guard.

Retry backoff. A requeued retry_safe job flips straight to QUEUED and is immediately re-claimable, so a deterministic failure retries back-to-back to max_attempts. Default workflows don't set retry_safe so it's narrow, but a small delay would help.

A few that look deliberate, want to confirm:

• One shared FIFO queue by created_timestamp, no fairness, so one caller flooding holds up everyone.
• No depth cap on the scaled queue, submit just persists a QUEUED row so it grows until retention or disk.
• No priority or isolation, so a slow flow class blocks faster ones behind it.

Minor: no single alembic head gate here, worth running alembic heads on the merge. And I haven't checked whether the started/completed counters double-count a retried job.

[ogabrielluiz]

Thanks, the terminal write is a real bug and you're right about the cause.

Fixed in 38a3ac9. Two deviations from your suggestion worth flagging:

Guarded on "not already terminal" rather than IN_PROGRESS. Several callers pass mark_in_progress=False and own the status themselves, so an IN_PROGRESS guard would have stranded their jobs non-terminal instead.

The owner guard is in, but an absent owner has to pass it. Owner is written by the heartbeat rather than at claim time, so a run that finishes before its first beat has no name on its row. My first version required a match and stranded exactly the fastest jobs. It checks "not somebody else's" rather than "provably mine". Losing the write logs a warning now instead of passing silently.

On the counters you hadn't checked: no double-count. retry_requeue_claim updates the same row and the counters count rows, so a retry counts once. The flip side is that retries are invisible, a job that failed twice then succeeded reads started=1, completed=1, failed=0. That probably wants its own metric.

Clock skew and retry backoff are both real and unfixed. Skew is inherent to comparing the reconciler's clock against a worker's timestamp, and I don't have better than requiring NTP and saying so. Backoff I'd like to do, it's small.

The three you asked about are deliberate, and they're in the description as the known ceiling with SKIP LOCKED as the marked upgrade path. The depth cap is the one I'd reconsider soonest, since growing until retention or disk is a worse failure than a rejected submit. Do you think that should come now rather than later?

alembic heads is single on the merge, c7d2b9e4f813. Agree it should be gated rather than left to habit.

[github-actions]

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft