Skip to content

Commit 085c681

Browse files
committed
docs(rfc): propose OpenShell testing strategy
Signed-off-by: Evan Lezar <elezar@nvidia.com>
1 parent 6bff09d commit 085c681

2 files changed

Lines changed: 361 additions & 51 deletions

File tree

‎CI.md‎

Lines changed: 2 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,60 +1,11 @@
11
# CI
22

3-
This document describes the current OpenShell continuous integration system,
4-
with a focus on what contributors need to do to get a pull request tested. It
5-
documents implemented workflow behavior rather than the complete desired test
6-
architecture.
3+
This document describes how OpenShell's continuous integration works for pull requests, with a focus on what contributors need to do to get their PR tested.
74

8-
For the target testing model and canonical local entry points, see
9-
[TESTING.md](TESTING.md). For pull request conventions, see
10-
[CONTRIBUTING.md](CONTRIBUTING.md).
5+
For local test commands see [TESTING.md](TESTING.md). For PR conventions see [CONTRIBUTING.md](CONTRIBUTING.md).
116

127
## Overview
138

14-
CI implements the testing layers defined in `TESTING.md` through separate
15-
workflow families:
16-
17-
| Testing responsibility | Current CI implementation |
18-
|---|---|
19-
| Static, build, unit, component, and SDK checks | `Branch Checks` across Linux x86_64, Linux ARM64, and macOS ARM64, with additional language-specific jobs |
20-
| Runtime and cross-component validation | Label-selected `Branch E2E Checks` for Docker, Podman, Kubernetes, VM, GPU, MCP, Python, and managed or standalone drivers |
21-
| Nix and `tmachine` validation | Release Dev builds exact-revision artifacts and test archives, then runs CLI conformance in disposable Ubuntu/Docker and Fedora/Podman guests |
22-
| Windows compatibility | Opt-in Windows MSVC x64 and ARM64 jobs; main and manual runs also build release binaries |
23-
| Integrated revision validation | Required statuses on merge-group SHAs before `main` advances |
24-
| Published artifact validation | Release workflows and post-publish release canaries |
25-
| Security analysis | Required dependency and deployment gates plus informational reports described below |
26-
27-
This is not yet the full target state. The project is migrating integration and
28-
Linux installation testing from `mise` tasks and workflow-local shell harnesses
29-
to Nix-built artifacts and `tmachine` wherever the runner can represent the
30-
environment. Today that path is limited to release conformance. `TESTING.md`
31-
defines the desired gates, matrices, migration phases, and deferred decisions.
32-
This document should describe a migration step as complete only after the
33-
workflow enforces it.
34-
35-
### Current tmachine implementation
36-
37-
`.github/workflows/conformance.yml` is the first implementation of this model.
38-
It downloads exact-revision binaries and OCI image artifacts, uses
39-
`nix run .#build-artifacts-test-archives` to package the test suite, and fans out
40-
`nix run .#tmachine -- test <scenario> <testsuite>` on KVM-enabled runners. The
41-
workflow caches prepared disks, while `tmachine` gives each test a fresh writable
42-
overlay. Release Dev currently calls the reusable workflow for these pairs:
43-
44-
| Scenario | Testsuite |
45-
|---|---|
46-
| `ubuntu-docker-rootful` | `conformance` |
47-
| `fedora-podman-rootful` | `conformance` |
48-
| `fedora-podman-rootless` | `conformance` |
49-
50-
This workflow currently runs from Release Dev. It is not a required pull-request
51-
or merge-queue gate. Branch E2E, GPU, Kubernetes, VM, Windows, and release-canary
52-
workflows continue to provide the coverage that has not migrated to tmachine.
53-
54-
The desired merge matrix, release-validation model, and migration sequence are
55-
defined in [TESTING.md](TESTING.md). They describe target behavior and do not
56-
become current CI behavior until the corresponding workflows implement them.
57-
589
PR CI that runs on NVIDIA self-hosted runners uses NVIDIA's copy-pr-bot. The bot mirrors trusted PR commits to internal `pull-request/<N>` branches in this repository. The gated workflows trigger on pushes to those branches, not on the original PR.
5910

6011
When a PR is not mirrored automatically, anyone with Write, Maintain, or Admin

0 commit comments

Comments
 (0)