feat(server): scope mutation locks and bound lock waits by EmilienM · Pull Request #4147 · NVIDIA/OpenShell

EmilienM · 2026-10-02T23:38:00Z

Summary

Scope gateway mutation locks by workspace and sandbox and bound every lock wait, so a burst of supervisor reconnects (rollout, scale-down, redirects) no longer queues behind one fleet-wide lock.

Related Issue

Part of #3528. Follows #3978, now merged. Follow-ups are tracked in #4307.

Changes

Replace the process-wide sync_lock and the single fleet-wide advisory key with global, workspace and sandbox lock scopes (shared/exclusive) held on a dedicated 4-connection PostgreSQL lock pool. Replicas still on the old release keep taking the global key, so a mixed-version rollout stays serialized.
Bound lock waits at 10 seconds. A timeout returns UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT and retry info. A lock connection PostgreSQL never opens returns INTERNAL, so it doesn't read as contention.
Reconcile endpoint status at startup one sandbox at a time, and skip the endpoint-status lock for a superseded supervisor session.
Sandbox settings are stored by sandbox name, which a per-sandbox lock doesn't protect once another sandbox reuses the name. A settings update now fails with NOT_FOUND or ABORTED instead of writing into the newer sandbox's settings.
Fix two races that exist on main, each in its own commit: DeleteProvider holds the mutation guard while it deletes, and staged provider credentials are released when a refresh fails early. On kind, the delete race left a dangling provider reference that crashlooped every gateway at startup.
Export lock wait and timeout metrics, add PostgreSQL integration tests run by mise run test:rust:postgres and a new Branch Checks job, and document lock-pool sizing. The reconnect-burst capacity check runs separately with mise run test:rust:postgres:bench, outside CI. Each pod now opens up to 14 PostgreSQL connections instead of 10, so raise max_connections before upgrading.

Testing

Rebased onto main after feat(server): place supervisor sessions by consistent hash #3661 merged. On the tip, mise run pre-commit and mise run docs:build:strict pass, mise run rust:lint and the openshell-server tests pass, and every commit builds with its tests.
mise run test:rust:postgres: 15 tests covering scope exclusion, interop with the old global key, cancellation, pool bounds and connection failures. mise run test:rust:postgres:bench passes on an idle machine.
kind with external PostgreSQL: the HA suite passed 3/3, the lock metrics showed up in Prometheus with zero timeouts, and a mixed fleet (old and new replicas serving together through upgrade, rollback and upgrade) ran about 11,000 mutations with no inconsistencies and no lock timeouts. These runs predate the rebases onto main; CI's HA E2E covers the current tip.

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Unit and PostgreSQL integration tests added
Operator documentation and the cluster debug skill updated

copy-pr-bot · 2026-10-02T23:38:05Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-10-06T18:43:49Z

Label test:e2e applied, but pull-request/4147 does not exist yet. A maintainer needs to comment /ok to test 71b9c5294ec18ced30b7179b177adeac54b3588f to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

github-actions · 2026-10-06T18:43:50Z

Label test:e2e-kubernetes applied, but pull-request/4147 does not exist yet. A maintainer needs to comment /ok to test 71b9c5294ec18ced30b7179b177adeac54b3588f to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

johntmyers · 2026-10-06T18:43:55Z

/ok to test 71b9c52

johntmyers · 2026-10-06T18:45:49Z

gator-agent

PR Review Status

The independent review of scoped mutation locks, bounded waits, and provider cleanup found no blocking findings. Current-head Branch Checks, Helm Lint, Trivy Changes, and E2E workflows are queued or running; Gator will monitor their results.

Blocking findings: None.
Carried findings: None.
Non-blocking suggestions: None.

Gator metadata

Validation: Implements the scoped mutation-lock work in feat(ha): add production scaling signals and graceful gateway redistribution #3528 and follows merged feat(server): add gateway capacity metrics and optional HPA #3978; no active duplicate found.
Docs: Fern HA connection sizing, mixed-version upgrades, lock metrics, and API error guidance updated; cluster debug skill updated.
Checks: DCO passes; no merge conflicts. Branch Checks (37513563884), Helm Lint (37513564067), and Trivy Changes (37512710449) are dispatched for the current head.
E2E: test:e2e and test:e2e-kubernetes applied. Branch E2E Checks (37513564920) is running after /ok to test refreshed the mirror; labels were present before dispatch. No rerun needed.
Local validation: PostgreSQL test-runner database selection test and diff whitespace check passed. Full Rust/PostgreSQL tests were not run locally because this sandbox lacks the toolchain and container engine.
Head SHA: 71b9c5294ec18ced30b7179b177adeac54b3588f
Base SHA: 8760396975f4b0f13cd6fd88ac1a46342ee84343
Merge base SHA: 8760396975f4b0f13cd6fd88ac1a46342ee84343
Patch ID: 10a548689cc4281ba946a4f7a503950e90537704
Gator payload: 10
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:watch-pipeline

johntmyers · 2026-10-06T22:15:05Z

/ok to test 51822ba

johntmyers · 2026-10-06T22:16:41Z

gator-agent

PR Review Status

The independent follow-up review found no blocking findings in the author’s changes across the rebase, including settings row identity checks, provider-name validation, SSH identity lock-pool separation, and lifecycle lock adaptations. The stale test mirror has been refreshed, and current-head Branch Checks, Helm Lint, Trivy Changes, and E2E workflows are running; Gator will monitor their results.

Blocking findings: None.
Carried findings: None.

Gator metadata

Validation: Continues the scoped mutation-lock work in feat(ha): add production scaling signals and graceful gateway redistribution #3528 and follows merged feat(server): add gateway capacity metrics and optional HPA #3978.
Docs: Fern HA guidance and the cluster debug skill cover the SSH identity lock behavior and connection sizing.
Checks: DCO passes; no merge conflicts. Branch Checks (37539382397), Helm Lint (37539382408), and Trivy Changes (37539113467) are running for the current head. Trivy workflow execution was approved after a narrowly scoped sandbox policy update.
E2E: test:e2e and test:e2e-kubernetes were present before mirror refresh. Branch E2E Checks (37539383455) is running; no label-help rerun request was posted.
Local validation: PostgreSQL test-runner database selection test and diff whitespace check passed. Full Rust/PostgreSQL tests could not run locally because this sandbox lacks the Rust toolchain and container engine.
Head SHA: 51822ba9c96ae3fad3cf94761e3f9f9a24dd5a90
Base SHA: 9a6148fc988a5e7a20a43805a52fdfaad766f8ae
Merge base SHA: 360c5a02cba5b9ee62f24e2528ab1ea2aead9b6f
Patch ID: c19aea2c1d6810b73eaed1fc917ee38092fff512
Gator payload: 10
Review mode: follow_up
Previous reviewed SHA: 71b9c5294ec18ced30b7179b177adeac54b3588f
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:watch-pipeline

FrostGod · 2026-10-07T08:08:23Z

+`OPENSHELL_REPLAY_TEST_DATABASE_URL` so legacy tests use that same database.
+Never point it at a database that a running gateway uses: the tests take
+fleet-wide advisory locks.
+CI does not run these tests; the Kubernetes HA e2e suite covers PostgreSQL end


Can we run the focused PostgreSQL lock tests in CI? They check cancellation, connection reuse, and compatibility with old replicas

Done. Branch Checks has a new "Rust PostgreSQL tests" job that runs the same runner script on a Linux runner, so the 15 postgres_* tests (cancellation, connection reuse, old-replica interop and the rest) run on every PR. The burst test is out of that suite now, see the other thread.

FrostGod · 2026-10-07T08:11:39Z

+        );
+        // Waits must stay far from the lock timeout, where requests fail.
+        assert!(
+            p99 * 5 < MUTATION_LOCK_TIMEOUT,


Can we move this latency assertion into a dedicated performance benchmark? In our earlier run, all 1,000 operations succeeded, but p99 was 5.35 seconds while other tests were running. The isolated rerun passed, so this threshold depends on machine load.

Agreed, that one is a capacity check, so it shouldn't gate anything. It's now bench_postgres_lock_pool_absorbs_a_12ms_reconnect_burst: test:rust:postgres and CI skip it, and mise run test:rust:postgres:bench runs it on its own with the p99 check. The repo has no benchmark harness and the test uses crate-private APIs, so this seemed like the simplest place for it.

FrostGod · 2026-10-07T08:12:21Z


        let mut sandbox_settings =
            load_sandbox_settings(state.store.as_ref(), &workspace, sandbox.object_name()).await?;
+        ensure_sandbox_keeps_name(state, &sandbox).await?;


Can we track the first settings write as a follow-up? A peer can delete the sandbox after this ownership check, and the first insert can still create settings inherited by a replacement using the same name. The row-ID check protects existing rows, but not the first insert.

Right, the row-ID check only covers rows that already exist. Main has the same race (lifecycle deletes don't take a database lock), so it's in #4307 with the other follow-ups.

FrostGod · 2026-10-07T08:13:20Z

+    /// Local S(global) S(workspace) X(sandbox), for provisioning-deadline
+    /// reconciliation, which re-derives configuration from provider and
+    /// profile records and must not interleave with their local writers.
+    pub(super) async fn lock_sandbox_local_in_workspace(


Please link a follow-up issue for bounding this wait and testing it. This path can hold the sandbox lock while waiting for a workspace lock, delaying start, stop, and delete for that sandbox.

Tracked in #4307, including a test with a sandbox key that sorts before its workspace key.

The openshell-server tests that need a real PostgreSQL server are ignored by default and had no shared way to run. The mutation-replay test reads its own OPENSHELL_REPLAY_TEST_DATABASE_URL, so every contributor had to provision a database by hand, and the advisory-lock tests that follow in this series need the same setup. Add mise run test:rust:postgres. It runs every ignored postgres_* test in openshell-server against OPENSHELL_TEST_POSTGRES_URL, or starts a disposable PostgreSQL container with Docker or Podman (CONTAINER_ENGINE selects one) on the image pinned by the Kubernetes e2e fixture and removes it on exit. Tests run one at a time because advisory locks are database-wide. The runner always points the legacy replay variable at the selected database, so an inherited URL cannot send that test to a different server. test:postgres-runner checks that selection with a fake cargo and needs no database or container engine. CI does not run the PostgreSQL tests; TESTING.md documents the task. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

DeleteProvider checked that no sandbox referenced the provider and then deleted the record without holding the sandbox mutation guard. Sandbox create and provider attach take that guard while they write provider references, so one of them could add a reference after the attached sandbox check passed, and the delete then removed a provider that a sandbox spec still named. Take the guard before the attached sandbox check, as provider create and update already do, so the check and the delete run against a stable set of sandbox references, and reject an empty name after authorization but before the guard. A new test holds the guard, attaches the provider while the delete waits, and asserts that the delete fails with FailedPrecondition and leaves the provider in place. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

A provider credential refresh stages the minted values under new credential handles before it commits the provider update, and deletes those handles when validation or persistence fails. Two earlier returns skipped that cleanup. The minted expiry was converted to a protobuf timestamp only after staging, so an expiry outside the timestamp range failed the refresh and left the staged values in the credential driver. A failure to acquire the sandbox mutation guard returned the same way. Convert the expiry before staging anything, so a bad value fails before any handle exists, and delete the staged handles when the guard cannot be acquired. A new test refreshes a stored credential with an expiry of i64::MAX and asserts that the call fails, the provider keeps its original handles, and the credential driver holds the same number of values as before. The guard failure path has no test here, because the SQLite guard used by unit tests cannot fail. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

When a supervisor session ends, the gateway resets that sandbox's tool server endpoint evidence unless a replacement session already owns endpoint observation. The replacement check ran only after the reset had read the sandbox and taken its sandbox mutation guard, so a reset whose supervisor had already reconnected still queued behind other mutations of that sandbox, and on PostgreSQL it used a lock-pool connection, only to find it had nothing to do. Check for a replacement first: a live local session, or a fresh owner record on a peer. When one exists, return without reading the sandbox or taking the guard. Otherwise take the guard and repeat the same check under it, because a replacement's pre-acknowledgement reset runs under the same sandbox key and resetting after it would wipe the replacement's fresh evidence. Both checks share one helper, so they cannot drift apart. When no replacement exists, the reset costs one extra owner-row read. This helps most when a replica notices a dead stream late, after the supervisor has already reconnected elsewhere (a network partition or a keepalive expiry). Deferring to a replacement leaves the old session's evidence to the replacement's pre-acknowledgement reset, which can time out on its bounded mutation lock wait. That reset's failure path removed the replacement and released its owner record but scheduled no invalidation, so the old session's success evidence could outlive both sessions. An old session that ended after its replacement registered already deferred this way, and the unguarded check now makes every disconnect that finds the replacement defer too. When the reset fails while the replacement is still current, also schedule the retried invalidation that follows a session end. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

A provider credential refresh deletes its staged handles when it cannot acquire the provider mutation guard, but that path had no test: the SQLite guard used by unit tests could not fail. Mutation locks now acquire under a deadline that tests can shorten, so the timeout path can be exercised directly. The new test shortens the lock deadline to 50 ms, holds the workspace mutation guard, and refreshes a provider that already stores an AWS access key. The refresh returns UNAVAILABLE with the MUTATION_LOCK_TIMEOUT reason, the provider keeps its original handle, which still resolves to the old key, and the credential driver holds the same number of values as before the refresh. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

A handler mutation guard can now wait up to 10 seconds for its keys and then fail with UNAVAILABLE, but nothing showed operators how long guards waited or how often they gave up, short of reading warning logs on each replica. Export openshell_server_mutation_lock_wait_seconds, a histogram of the time one guard took to acquire all of its keys (local registry plus PostgreSQL advisory locks), recorded on success, and openshell_server_mutation_lock_timeouts_total, a counter of guard acquisitions that timed out. Both carry a bounded scope label: global, workspace or sandbox. The platform scope, whose workspace is empty, counts as global. The histogram uses the same 1 ms to 15 s buckets as the relay and peer histograms, and the timeout counter starts at 0 for every scope, so rate() works on an idle replica. The timeout warning now also logs the scope. A guard that fails for another reason, such as a lock connection PostgreSQL did not open, is not counted as a timeout and logs a "mutation lock acquisition failed" warning instead. Lifecycle, driver-watch and reconcile paths take only process-local locks and never time out, so they are not recorded. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

The scoped mutation locks were only unit-tested on SQLite, where the database layer is a no-op. Add ignored postgres_* tests that run the advisory-lock layer against a real PostgreSQL through mise run test:rust:postgres, each in its own disposable schema (TestSchema). persistence/mutation_lock_pg_tests.rs uses two stores on one database as two replicas. It covers disjoint sandbox scopes holding at once, a workspace key blocking only sandboxes in that workspace, the global key blocking every scope, and exclusion in both directions against a raw holder of the legacy global key, whose 55P03 stays a database error outside a guard acquisition. It checks that timed-out and cancelled acquisitions release the keys they already took, that a released lock connection returns to the pool without residual locks, that a release stalled on the network closes its session and the pool recovers, and that the lock pool never grows past its size. It also checks that a full lock pool times out as lock contention, while a lock connection that PostgreSQL never answers fails as a database error. compute/mutation_guard.rs runs the random scope mix across two PostgreSQL runtimes and checks that a by-id guard waits for a workspace writer on the other replica. A bench_postgres_* capacity check measures a paced burst of 500 reconnects 12 ms apart into one replica and fails when the p99 lock wait reaches a fifth of the lock timeout. Machine load moves those waits, so mise run test:rust:postgres skips it and mise run test:rust:postgres:bench runs it alone, through a new OPENSHELL_TEST_POSTGRES_PREFIX setting in the runner. The tests add test-only helpers: the lock pool's size and idle count and the backend pid of a held lock session. Nothing changes outside tests and the test runner. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

Branch Checks now runs the postgres_* tests through the same runner developers use, on an x86_64 Linux runner with Docker. They cover lock scope exclusion, interop with replicas still on the old global key, cancellation, lock-pool bounds and connection failures, which no other CI lane checks directly. The load-sensitive bench_postgres_* capacity checks stay local. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

In the High Availability guide, size PostgreSQL for 14 connections per gateway pod, 10 for data and 4 for the new lock pool, so three or more replicas now exceed what the PostgreSQL default leaves, and keep headroom for lock sessions that cancelled requests leave behind. Explain which changes lock a sandbox, a workspace, or the whole fleet, and that a lock wait over 10 seconds returns a retryable UNAVAILABLE with reason MUTATION_LOCK_TIMEOUT. A new upgrade subsection says to raise max_connections to the 14-per-pod size before upgrading, because the rollout already runs new pods at that count, and warns that older replicas keep serializing every mutation until the rollout finishes. Monitor Capacity gains the lock metrics and their queries, and names slow driver, credential, middleware, or profile source calls as another cause of lock waits. List the lock wait histogram and timeout counter, with their scope label, on the Gateway Metrics page. The counter also counts background work and leaves out lock connections that PostgreSQL does not open although at least a second of the wait remained. In the cluster debugging skill, replace the single advisory-lock paragraph with the lock levels and pool, what each timeout detail means, how a lock connection that PostgreSQL does not open shows up, when terminating a lock backend is safe, a pg_locks query that shows holders and waiters, and how to spot the global key. The per-pod greps also read the timeout counter, and new rows cover lock timeouts and lock connections that PostgreSQL does not open. Part of NVIDIA#3528 Signed-off-by: Emilien Macchi <emacchi@redhat.com>

johntmyers · 2026-10-07T16:40:48Z

/ok to test c00b632

johntmyers · 2026-10-07T16:43:45Z

gator-agent

PR Review Status

Thanks @EmilienM. I checked your update adding the focused PostgreSQL tests to Branch Checks and separating the load-sensitive reconnect benchmark; the independent follow-up review found no blocking findings in the author-only delta. I also checked that the first-settings-write and provisioning lock-wait concerns you acknowledged are tracked in #4307; unchanged follow-up work does not block this PR.

The current-head test mirror has been refreshed, and Branch Checks, Helm Lint, Trivy Changes, and E2E are queued or running. Gator will inspect their results next cycle.

Blocking findings: None.
Carried findings: None.

Gator metadata

Validation: Continues the scoped mutation-lock work in feat(ha): add production scaling signals and graceful gateway redistribution #3528 and follows merged feat(server): add gateway capacity metrics and optional HPA #3978.
Docs: Fern HA connection sizing, mixed-version upgrades, lock metrics, and API errors remain documented; the cluster debug skill is updated. TESTING.md now describes PostgreSQL CI coverage and the separate benchmark.
Checks: DCO passes; no merge conflicts. Current-head Branch Checks (37653952709), Helm Lint (37653952804), and Trivy Changes (37653245612) are dispatched. Trivy workflow execution was approved after an exact-run sandbox policy update.
E2E: test:e2e and test:e2e-kubernetes remain applied. Branch E2E Checks (37653954219) is dispatched after /ok to test refreshed the mirror. No label-help rerun request was posted.
Local validation: PostgreSQL runner database and prefix selection tests and diff whitespace check passed. Full Rust/PostgreSQL tests could not run locally because this sandbox lacks the Rust toolchain and container engine; CI includes the PostgreSQL tests.
Head SHA: c00b63248dea0b0dc9020959ae0eb440cb4e3358
Base SHA: e406c2befbe65501409026d49259602669f7246a
Merge base SHA: e406c2befbe65501409026d49259602669f7246a
Patch ID: 795af6ac39b051a98546eca7c77e8d8fd621c496
Gator payload: 10
Review mode: follow_up
Previous reviewed SHA: 51822ba9c96ae3fad3cf94761e3f9f9a24dd5a90
Review budget exhausted: no
Maintainer decision required: no
Next state: gator:watch-pipeline

EmilienM · 2026-10-07T19:27:17Z

@johntmyers I think the CI failures are unrelated can you please /ok to test again?
Thanks!

johntmyers · 2026-10-07T19:35:55Z

/ok to test c00b632

Signed-off-by: divesh <dgude@nvidia.com>

johntmyers · 2026-10-08T08:54:52Z

/ok to test 7663cc2

johntmyers · 2026-10-08T08:56:32Z

gator-agent

PR Review Status

Thanks @EmilienM. I checked your request to refresh testing and posted /ok to test for the latest head; the test mirror now matches it. The independent Critical-only review of the author-only merge resolution found no new Critical defects, and no carried blocking findings remain.

Current-head Branch Checks, Helm Lint, Trivy Changes, and E2E are queued or running. Gator will inspect their results next cycle.

Blocking findings: None.
Carried findings: None.

Gator metadata

Validation: Continues scoped mutation-lock work in feat(ha): add production scaling signals and graceful gateway redistribution #3528 and follows merged feat(server): add gateway capacity metrics and optional HPA #3978; this head resolves the merge with main without adding a new subsystem or public configuration surface.
Docs: Existing Fern HA connection sizing, mixed-version upgrade, lock metric, and API error guidance remains applicable; the merge resolution adds no new user-facing contract.
Checks: DCO passes; no merge conflicts. Current-head Branch Checks (37753030675), Helm Lint (37753030964), and Trivy Changes (37751905835) are dispatched. Trivy workflow execution was authorized after a policy proposal scoped to that exact run was approved and reloaded.
E2E: test:e2e and test:e2e-kubernetes remain applied. Current-head Branch E2E Checks (37753032025) is queued after mirror refresh. No new label-help rerun instruction was posted.
Local validation: PostgreSQL runner database and prefix selection tests and diff whitespace check passed. Full Rust/PostgreSQL tests were not run locally because this sandbox lacks the toolchain; CI includes PostgreSQL integration coverage.
Head SHA: 7663cc225a538f1bbfd925f8cba367d8e51e10d0
Base SHA: 67adcf1a3cf054be54dfdbf6a44c65b4d426b1ec
Merge base SHA: 67adcf1a3cf054be54dfdbf6a44c65b4d426b1ec
Patch ID: 2fb46059e80b40e41aaf7379058479a41dc98084
Gator payload: 10
Review mode: critical_only
Previous reviewed SHA: c00b63248dea0b0dc9020959ae0eb440cb4e3358
Review budget exhausted: yes
Maintainer decision required: no — no open obligations, qualifying scope growth, or proposed Warning requires a decision.
Next state: gator:watch-pipeline

EmilienM mentioned this pull request Oct 5, 2026

feat(server)!: harden gateway peer transport and expand HA conformance #3825

Open

7 of 8 tasks

EmilienM force-pushed the feat/3528-scoped-locks/EmilienM branch from 26c5f0d to 71b9c52 Compare October 6, 2026 18:37

EmilienM marked this pull request as ready for review October 6, 2026 18:37

EmilienM requested review from a team, derekwaynecarr, mrunalp and sjenning as code owners October 6, 2026 18:37

johntmyers added test:e2e Requires end-to-end coverage test:e2e-kubernetes Requires Kubernetes end-to-end coverage labels Oct 6, 2026

EmilienM force-pushed the feat/3528-scoped-locks/EmilienM branch from 71b9c52 to 51822ba Compare October 6, 2026 22:12

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status labels Oct 6, 2026

FrostGod reviewed Oct 7, 2026

View reviewed changes

EmilienM added 3 commits October 7, 2026 11:19

EmilienM added 6 commits October 7, 2026 11:40

EmilienM force-pushed the feat/3528-scoped-locks/EmilienM branch from 51822ba to c00b632 Compare October 7, 2026 16:35

EmilienM mentioned this pull request Oct 7, 2026

feat(server): follow-ups to scoped mutation locks (#4147) #4307

Open

2 of 10 tasks

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:blocked Gator is blocked by process or repository gates and removed gator:approval-needed Gator completed review; maintainer approval needed gator:watch-pipeline Gator is monitoring PR CI/CD status labels Oct 7, 2026

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status and removed gator:blocked Gator is blocked by process or repository gates labels Oct 7, 2026

EmilienM mentioned this pull request Oct 7, 2026

feat(ha): add production scaling signals and graceful gateway redistribution #3528

Open

7 tasks

fix(server): resolve main merge conflict

7663cc2

Signed-off-by: divesh <dgude@nvidia.com>

johntmyers added gator:watch-pipeline Gator is monitoring PR CI/CD status gator:approval-needed Gator completed review; maintainer approval needed and removed gator:blocked Gator is blocked by process or repository gates gator:watch-pipeline Gator is monitoring PR CI/CD status labels Oct 8, 2026

Conversation

EmilienM commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

johntmyers commented Oct 6, 2026

Uh oh!

johntmyers commented Oct 6, 2026

PR Review Status

Uh oh!

johntmyers commented Oct 6, 2026

Uh oh!

johntmyers commented Oct 6, 2026

PR Review Status

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johntmyers commented Oct 7, 2026

Uh oh!

johntmyers commented Oct 7, 2026

PR Review Status

Uh oh!

EmilienM commented Oct 7, 2026

Uh oh!

johntmyers commented Oct 7, 2026

Uh oh!

johntmyers commented Oct 8, 2026

Uh oh!

johntmyers commented Oct 8, 2026

PR Review Status

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

EmilienM commented Oct 2, 2026 •

edited

Loading