Skip to content

test(tmachine): use K3s ClusterIP, wait for gateway, add failure diagnostics - #4258

Merged
matthewgrossman merged 3 commits into
mainfrom
test/4254-k3s-clusterip/matthewgrossman
Oct 9, 2026
Merged

matthewgrossman merged 3 commits into
mainfrom
test/4254-k3s-clusterip/matthewgrossman

Conversation

@matthewgrossman

@matthewgrossman matthewgrossman commented Oct 6, 2026 •

Copy link
Copy Markdown
Member

Summary

Connect tmachine K3s conformance clients directly to the gateway's ClusterIP Service, removing the long-lived kubectl port-forward process. The CLI runs on the K3s node inside the guest, so no external exposure is needed.

This PR also adds two net-new behaviors:

  • Shared gateway wait (all installers). Before conformance, the shared conformance playbook waits up to five minutes for openshell status to report a connected gateway. Every test boot restarts the gateway; installers that are already up pass immediately. If the gateway never connects, conformance is skipped, diagnostics are collected, and the run fails with a clear message instead of per-scenario preflight failures.
  • Failure diagnostics. On conformance or gateway-wait failure, one openshell_diagnostics role collects the systemd gateway journal (moved from cli.yaml) and, when K3s is installed, bounded K3s diagnostics: K3s journal, API readiness, nodes, gateway Pod state, Services, EndpointSlices, iptables Service rules (nft and legacy), gateway events and current/previous logs, and CLI status. Integration Tests uploads the K3s diagnostics as tmachine-diagnostics-* artifacts.

Related Issue

Closes #4254. Replaces the closed NodePort draft #4255.

Changes

  • openshell-k3s.yaml: replace the port-forward unit with ClusterIP and gRPC port discovery, registered once through the existing openshell_client role. Recreating the Service requires reprovisioning the fixture.
  • conformance/cli.yaml: add the shared gateway wait; replace inline gateway-log tasks with the shared diagnostics role behind a single conformance_failed condition.
  • roles/openshell_diagnostics/: new shared diagnostics role and K3s collection script.
  • integration-runner.yml: upload diagnostics artifacts.
  • CI.md: short note on the K3s lane, the shared wait and diagnostics.

No chart, runtime or tmachine Rust changes. Install-time diagnostics are intentionally out of scope; Helm install failures surface through normal Ansible output.

Testing

  • Ansible syntax checks (openshell-k3s.yaml, conformance/cli.yaml, conformance/policy-advisor.yaml) and shellcheck pass.
  • Local Ansible logic check of the shared wait with a stub CLI: connected + passing tests runs clean; connected + failing tests collects diagnostics and fails on conformance; never-connected skips conformance, collects diagnostics and fails on the gateway wait.
  • ClusterIP jsonpath verified against a sample Service (10.43.206.224:8080; headless None:8080 rejected).
  • Earlier local tmachine K3s validation (ARM64 QEMU/HVF) of the ClusterIP transport passed 8/8 conformance tests and kept the registration across gateway Pod replacement. That run predates the shared wait and diagnostics restructure.
  • Branch E2E run 37968718738 on head 1767ec0 passed, including all four conformance integration lanes (ubuntu-k3s, deb, rpm rootful, rpm rootless) and the feature- and driver-specific lanes. Failure diagnostics do not run on a passing run; the failure path is covered by the local stub check above.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Relevant CI guidance updated

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@matthewgrossman
matthewgrossman marked this pull request as ready for review October 9, 2026 16:34
Replace the K3s-specific boot readiness hook with one shared gateway wait in the conformance playbook. Skip conformance and collect the existing diagnostics when the gateway never connects. Discover the gateway Service port with its ClusterIP, and capture both iptables backends in K3s diagnostics.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman added the test:e2e Requires end-to-end coverage label Oct 9, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test b14566a

@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown

Label test:e2e applied for b14566a. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Comment thread .agents/skills/watch-github-actions/SKILL.md Outdated
Comment thread .github/workflows/integration-runner.yml
Comment thread tests/ansible/playbooks/conformance/cli.yaml Outdated
Comment thread tests/ansible/playbooks/conformance/cli.yaml Outdated
Comment thread tests/ansible/playbooks/openshell-k3s.yaml Outdated
Comment thread CI.md Outdated
Move failure diagnostics into one openshell_diagnostics role that collects the systemd gateway journal and, when K3s is installed, the K3s diagnostics. Drop install-time diagnostics so the K3s installer change stays minimal, revert unrelated skill edits, and shorten the CI.md note.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
@matthewgrossman matthewgrossman changed the title test(k3s): use direct ClusterIP access in tmachine test(tmachine): use K3s ClusterIP, wait for gateway, add failure diagnostics Oct 9, 2026
@matthewgrossman

Copy link
Copy Markdown
Member Author

/ok to test 1767ec0

@matthewgrossman
matthewgrossman added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit d789ec6 Oct 9, 2026
113 checks passed
@matthewgrossman
matthewgrossman deleted the test/4254-k3s-clusterip/matthewgrossman branch October 9, 2026 20:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test: remove port-forward dependency from tmachine K3s conformance

2 participants