bug: Podman driver picks the wrong host IP for the supervisor callback on multi-homed Linux hosts (regression from #2942) #3412

Activity

1. github-actions

   added

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Sep 17, 2026
2. jgarciao commented on Sep 17, 2026

   

   jgarciao

   on Sep 17, 2026

   ContributorAuthor

   More actions

   CC: @elezar

   👀React with 👀1Reacted by Evan Lezar
3. elezar

   added

   state:validated

   area:compute

   topic:networking

   os:linuxIssue affects Linux hostsIssue affects Linux hosts

   and removed

   state:triage-neededOpened without agent diagnostics and needs triageOpened without agent diagnostics and needs triage

   on Sep 17, 2026
4. elezar commented on Sep 17, 2026

   

   elezar

   on Sep 17, 2026

   Member

   More actions

   📋 triage-agent

   Triage Assessment

   Classification: validated-bug

   Summary

   Issue #3412 is a confirmed regression in the local Linux Podman callback path. The report provides a before/after version bisection, concrete listener and resolver addresses, supervisor logs, and a discriminating loopback workaround. Current main retains the reported mismatch, so the evidence is sufficient to validate the bug without requesting more information.

   Investigation

   PR #2942 moved the callback-capable supervisor into the host network namespace. However, the default Podman endpoint still uses host.containers.internal, an empty host_gateway_ip still injects Podman's host-gateway mapping, and rootless listener negotiation still selects the host default-route interface. On a multi-homed host, Podman's independently selected host-gateway address can therefore differ from the gateway listener address, exactly as reported.

   The host_gateway_ip = "127.0.0.1" workaround is consistent with the current architecture: the supervisor shares the host network namespace, the gateway already listens on loopback, and the generated certificate covers the alias and loopback address. The workaround is not an adequate default because it depends on non-obvious topology details and the surfaced workload exit does not identify the supervisor callback failure.

   PR #2874 is related but does not address this regression. It handles a rootful Podman startup failure where an exact managed-bridge address is not yet locally bindable. Issue #3412 occurs after listeners bind successfully, in rootless mode, because the host-networked supervisor resolves host-gateway to a different interface. PR #2874 does not change callback endpoint selection, host entries, or supervisor failure reporting.

   No duplicate was found. Issue #2540 is adjacent rootless slirp4netns relay work based on a callback originating in an isolated rootless network namespace; #3412 concerns the post-#2942 host-networked supervisor. Issues #1952, #1519, #1634, and #1909 cover different ownership, macOS, or containerized-gateway cases.

   The misleading ContainerExited: code 0 result is also consistent with current watcher behavior: when the supervisor exits, reconciliation stops and re-inspects the workload, then reports the workload's exit state rather than the supervisor's policy-fetch failure.

   Impact Signals

   • Affected users/scope: Linux users of the local Podman driver on multi-homed hosts; rootless Fedora 44 with Podman 5.8.4 is confirmed. The rootful breadth is not yet measured.
   • Regression: Yes; reported absent in 0.0.116 and introduced by PR feat(isolation): implement the RFC 0012 sandbox architecture #2942. Current main retains the relevant behavior.
   • Workaround: Available by setting host_gateway_ip = "127.0.0.1", but topology-specific, non-obvious, and obscured by misleading diagnostics.
   • Evidence quality: High. The report includes a version bisection, exact reproduction commands and addresses, logs, a working discriminating override, and code-level causal alignment. The triage did not independently recreate the multi-homed Fedora environment.

   Human Decision Required

   Decide whether OpenShell should address this issue. If yes, apply
   state:accepted, associate it with a roadmap item, or do both, and decide
   whether the work remains human-owned. Either action records acceptance;
   roadmap placement additionally records sequencing.
   To queue investigation or planning for an unattended agent, also apply
   agent:plan-requested. You can instead directly ask an agent to use
   create-spike or build-from-issue on this issue; the agent will warn about
   missing expected workflow labels and continue without changing them. If no,
   close it as not planned and record the rationale.
5. elezar commented on Sep 17, 2026

   

   elezar

   on Sep 17, 2026

   Member

   More actions

   🏗️ build-plan

   Implementation Plan

   Issue type: fix
   Complexity: Medium
   Confidence: High — the callback topology after PR #2942 is clear from the current code

   Summary

   Update the existing PR #2874 implementation so callback routing follows the callback producer's network namespace rather than Podman's rootful/rootless mode. Native Linux supervisors use the host network namespace, so automatic callbacks will use the existing gateway aliases mapped to host loopback and request a loopback listener; Podman Machine and explicit remote endpoints keep their platform-specific routes.

   Scope

   • crates/openshell-driver-podman/src/driver.rs: select loopback for automatic Linux callbacks, remove bridge-gateway discovery and the rootless network-helper gate from this path, preserve explicit remote endpoints and explicit host-gateway overrides.
   • crates/openshell-driver-podman/src/container.rs: map the supervisor's driver-owned aliases to 127.0.0.1 for the automatic Linux host-network route.
   • crates/openshell-driver-podman/src/config.rs: document the updated empty/default host_gateway_ip behavior.
   • proto/compute_driver.proto, crates/openshell-server/, and Docker driver compatibility changes: remove PR fix(gateway)!: handle delayed Podman bridge listeners #2874's delayed-bind protocol/server machinery because the managed bridge is no longer the callback origin after PR feat(isolation): implement the RFC 0012 sandbox architecture #2942.
   • architecture/gateway.md, crates/openshell-driver-podman/NETWORKING.md, docs/reference/gateway-config.mdx, docs/reference/sandbox-compute-drivers.mdx, and skills/debug-openshell-cluster/SKILL.md: replace delayed-bind guidance with the callback-origin invariant and deployment boundaries.

   Implementation Steps

   1. Remove the delayed-bind protobuf/server implementation and restore the existing exact-address API.
   2. Make automatic Linux Podman callbacks request loopback for both rootful and rootless operation; retain explicit remote and Podman Machine behavior.
   3. Map automatic Linux supervisor aliases to loopback and add serialized-spec regression coverage.
   4. Update architecture, configuration, compute-driver, and troubleshooting documentation.
   5. Run focused Podman driver and gateway-listener tests, repository pre-commit checks, and the existing Fedora rootless/rootful validation paths where available.

   Test Plan

   • Unit tests: Cover rootful/rootless/slirp automatic loopback selection, explicit host_gateway_ip, explicit remote endpoints, loopback host entries, and absence of managed-network gateway inspection.
   • Integration tests: Run existing gateway listener tests for primary-listener reuse and callback-only loopback binding.
   • E2E tests: Reuse the existing Fedora rootless and rootful conformance paths without adding CI or conformance tooling to PR fix(gateway)!: handle delayed Podman bridge listeners #2874.

   Risks & Open Questions

   • Automatic Linux loopback assumes a native co-located gateway and Podman host. Containerized or remote gateways must configure an explicitly routable grpc_endpoint and appropriate gateway exposure.
   • Podman Machine must retain its gvproxy route because supervisor loopback is inside the VM.
   • The misleading ContainerExited: code 0 diagnostic requires an explicit supervisor-derived terminal condition and possibly structured failure propagation. Keep that lifecycle/diagnostics work separate from PR fix(gateway)!: handle delayed Podman bridge listeners #2874; the updated PR fixes the callback regression but does not claim that diagnostic criterion.

   Documentation Impact

   • Update docs/reference/gateway-config.mdx because the effective default semantics of empty host_gateway_ip change on Linux.
   • Update the Podman compute-driver, architecture, networking, and troubleshooting documentation listed above.
   • No SELinux or AppArmor-specific behavior is introduced.

   ---

   Revision 1 — initial plan for updating PR #2874
6. politerealism commented on Sep 17, 2026

   

   politerealism

   on Sep 17, 2026

   Contributor

   More actions

   I'm picking this up.
7. jgarciao commented on Sep 18, 2026

   

   jgarciao

   on Sep 18, 2026

   ContributorAuthor

   More actions

   Verified fixed by #3365 ✅

   Built the gateway + runtime images from #3365 and re-ran sandbox create with default config (no host_gateway_ip) on a genuinely multi-homed host — wired dock (enp0s13f0u2, default route metric 100), Wi-Fi (wlp9s0f0, metric 600), and Tailscale (wt0) all up. The sandbox reaches Ready.

   Why #3365 fixes it

   The fix changes how the Podman driver picks the supervisor's gateway endpoint. Since RFC 0012 the supervisor runs host-networked (--network host), so it shares the host's loopback. #3365 adds select_grpc_endpoint(), which on a Linux host returns http://127.0.0.1:<gateway-port> instead of the host.containers.internal alias:

   // PodmanEndpointEnvironment
   LinuxHost     => "127.0.0.1",
   PodmanMachine => "host.containers.internal",

   Because the endpoint is now loopback, Podman's host-gateway resolution is off the callback path entirely — it no longer matters which interface Podman would have mapped host-gateway to (the wired IP, or the Tailscale IP as in the original report). That's the root cause removed, not worked around.

   The old callback machinery this issue described — GetGatewayListenerRequirements negotiation, the DefaultRouteInterface listener, and nonlocal bind support — is deleted in the same PR.

   Verification evidence

   # gateway (driver) log
   INFO openshell_driver_podman::driver: Auto-detected gRPC endpoint grpc_endpoint=http://127.0.0.1:18080 tls=false
   
   # supervisor container env
   OPENSHELL_ENDPOINT=http://127.0.0.1:18080
   
   # supervisor container: Health=healthy, no "Policy fetch failed" lines
   

   For contrast, a main build on the same host hands the supervisor OPENSHELL_ENDPOINT=http://host.containers.internal:18080; that only works when Podman's host-gateway happens to resolve to the interface the gateway binds, which is exactly the fragile behavior this issue is about.

   Acceptance criteria

   • Multi-homed Linux host reaches Ready with default config (no manual host_gateway_ip).
   • Supervisor reaches the gateway regardless of Podman's host-gateway resolution (loopback removes that dependency).
   • Covered by a test — grpc_endpoint_uses_loopback_on_linux asserts the Linux endpoint selection.
8. politerealism commented on Sep 21, 2026

   

   politerealism

   on Sep 21, 2026

   Contributor

   More actions

   Looks like we can close this issue then @elezar
9. johntmyers

   closed this as completedon Sep 21, 2026