Skip to content

feat(server)!: harden gateway peer transport and expand HA conformance - #3825

Open
EmilienM wants to merge 7 commits into
NVIDIA:mainfrom
EmilienM:feat/3529-ha-peer-transport/EmilienM
Open

EmilienM wants to merge 7 commits into
NVIDIA:mainfrom
EmilienM:feat/3529-ha-peer-transport/EmilienM

Conversation

@EmilienM

@EmilienM EmilienM commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Gateway replicas on PostgreSQL relay sandbox sessions to each other, authenticated with a ServiceAccount token. Until now that traffic could go over plaintext with just a warning, or trust any platform root CA. This PR requires HTTPS with a pinned peer CA, and adds an HA e2e that runs real sandbox operations through a replica that doesn't own the session while the gateway scales, loses the owner pod, and rolls.

Related Issue #3529

Changes

  • Peers dial only https:// and trust only OPENSHELL_PEER_TLS_CA_FILE, never platform roots. At startup the gateway checks the CA, the server name, and its own certificate. Plaintext peers need OPENSHELL_PEER_ALLOW_INSECURE_TRANSPORT=true on a plaintext gateway, which then logs a warning on every start.
  • A relay to an owner that the local policy refuses now fails immediately instead of retrying for 15 seconds.
  • Helm refuses to render a PostgreSQL release with server.disableTls=true unless server.peer.allowInsecureTransport=true is set, and always wires the peer CA when TLS is on.
  • New kubernetes_ha_operations e2e. One sandbox lives through scale-up, scale-down, graceful and forced owner loss, and a rolling restart. After each step, exec, file transfer, forwards, and policy and provider updates go through a non-owner replica. The new e2e:kubernetes:ha-tls lane runs it with HTTPS peers.
  • Docs: peer transport settings in the gateway configuration reference, "Secure Peer Transport" and "Supported Operations" in the HA guide, and debug skill rows for the new errors.

Breaking (PostgreSQL only): releases with server.disableTls=true need server.peer.allowInsecureTransport=true, custom server TLS Secrets need ca.crt, and non-chart HTTPS peers need OPENSHELL_PEER_TLS_CA_FILE. Upgrade steps are in the HA guide.

Testing

  • Lint, server tests (including 55 new peer transport tests), Helm tests, and docs checks pass.
  • The new e2e passed on a local kind cluster (rootless Podman) in both the HTTPS and plaintext lanes at an earlier revision. I haven't re-run it since the rebase. The HA lanes need the test:e2e-kubernetes label.
  • Checks appropriate to the affected code and behavior pass
  • Unit tests added/updated (if applicable)
  • E2E tests added/updated (if applicable)

Checklist

Gemini_Generated_Image_ypqi10ypqi10ypqi

@copy-pr-bot

copy-pr-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@johntmyers johntmyers assigned johntmyers and drew and unassigned johntmyers Sep 29, 2026
@EmilienM
EmilienM force-pushed the feat/3529-ha-peer-transport/EmilienM branch 2 times, most recently from 921abb6 to 0a11343 Compare September 29, 2026 18:38
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
…peer CA

Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>
Signed-off-by: Emilien Macchi <emacchi@redhat.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants