Skip to content

ci: align sublibrary CI to OrdinaryDiffEq canonical translating dispatcher - #953

Closed
ChrisRackauckas-Claude wants to merge 3 commits into
SciML:masterfrom
ChrisRackauckas-Claude:claude/sublibrary-test-groups
Closed

ChrisRackauckas-Claude wants to merge 3 commits into
SciML:masterfrom
ChrisRackauckas-Claude:claude/sublibrary-test-groups

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Jun 4, 2026 •

Copy link
Copy Markdown
Member

Summary

This PR aligns the NonlinearSolve sublibrary CI to the exact OrdinaryDiffEq.jl canonical pattern: a translating root dispatcher plus clean-env sublibrary runtests, using the standard Core / QA / GPU test groups.

It keeps the centralized sublibrary CI form merged in #951 — .github/workflows/SublibraryCI.yml continues to call SciML/.github/.github/workflows/sublibrary-tests.yml@v1, and no bespoke CI_<sublib>.yml is restored. No workflow files are changed.

The canonical pattern (matching OrdinaryDiffEq)

The reusable sublibrary-tests.yml@v1 runs Pkg.test on the root package with env GROUP set to <sub> (Core) or <sub>_<Section> (e.g. <sub>_QA). The reference (SciML/OrdinaryDiffEq.jl test/runtests.jl) handles this by having the root runtests detect a sublibrary GROUP, activate lib/<sub>, translate the suffix to a clean section name, set it as a distinct env var, and Pkg.test the sublibrary. The sublibrary runtests reads only that clean env var (e.g. OrdinaryDiffEqRosenbrock reads just get(ENV, "ODEDIFFEQ_TEST_GROUP", "ALL")).

Earlier revisions of this PR instead put a per-package prefix-strip shim (_G/_SUB/startswith) in every sublibrary runtests and left the root runtests with no sublibrary dispatcher — so the sublibrary tests never actually ran under SublibraryCI. This revision adopts the OrdinaryDiffEq design exactly.

What changed

Root test/runtests.jl — canonical translating dispatcher prelude

Added at the top, reading the raw GROUP env before the main-package logic lowercases/parses it:

  • _detect_sublibrary_group (copied verbatim from OrdinaryDiffEq): isdir(lib/<group>) → (group, "Core"), else scan underscores right-to-left for the longest lib/<prefix> match → (prefix, suffix), else (group, "Core").
  • If base_group is a lib/<sub> directory: Pkg.activate(lib/<sub>), on Julia < 1.11 transitively Pkg.develop the activated project's [sources] path deps (verbatim OrdinaryDiffEq block, never developing the active project itself), then withenv("NLS_TEST_GROUP" => test_group) run Pkg.test(base_group; julia_args=["--check-bounds=auto","--depwarn=yes"], force_latest_compatible_version=false, allow_reresolve=true).
  • Otherwise (all/downstream/nopre/bounds/cuda/wrappers/trim/…): control falls through to the existing main-package logic, preserved byte-for-byte in the else branch (its TEST_ARGS parsing, lowercase(GROUP), Enzyme/Mooncake/CUDA EXTRA_PKGS, worker config, ReTestItems.runtests, and trim activation are unchanged).

Each lib/<sub>/test/runtests.jl — clean-env reader, no shim

The _G/_SUB/startswith prefix-strip shim is replaced by:

const GROUP = get(ENV, "NLS_TEST_GROUP", "All")

The standard Core/QA(/GPU via cuda) dispatch in each file is otherwise unchanged — functional items run under Core/All, Aqua/JET/ExplicitImports/alloc under QA/All, CUDA under GPU/cuda. (SimpleNonlinearSolve's now-unused parse_test_args/get_from_test_args_or_env helpers, which only fed the GROUP read, are dropped in favor of the plain get(ENV, …).) This matches OrdinaryDiffEqRosenbrock/test/runtests.jl.

The pre-existing @itesitem → @testitem fix in SimpleNonlinearSolve/test/core/allocation_tests.jl (so the QA allocation test actually runs) is retained.

Standard groups

  • Core — all functional/correctness tests (solver tests, adjoint, wrappers, basic, AD, …). Default-on group.
  • QA — Aqua + ExplicitImports (+ JET / allocation where present).
  • GPU — CUDA tests, handled by the separate unchanged GPU.yml workflow (which sets GROUP="cuda" on a self-hosted GPU runner).

No test_groups.toml in any sublibrary: with none present, the reusable compute_affected_sublibraries.jl falls back to its standard defaults — Core on [lts, 1.11, 1, pre] + QA on [1], all ubuntu-latest — emitting GROUP="<sub>" (Core) and GROUP="<sub>_QA" (QA). CUDA is covered by GPU.yml, so no [GPU] section is needed.

Verification (run locally, no heavy test runs)

  • (a) _detect_sublibrary_group("<sub>") == ("<sub>", "Core") and _detect_sublibrary_group("<sub>_QA") == ("<sub>", "QA") for all 10 sublibraries — all OK.
  • (b) For each sublibrary runtests, NLS_TEST_GROUP="Core" enters the Core block and NLS_TEST_GROUP="QA" enters the QA block (non-empty in each case; SciPy intentionally has no QA items, documented in-file) — all OK.
  • (c) For the main package, GROUP values nopre/cuda/all/downstream/bounds/wrappers/trim are not detected as sublibraries (isdir(lib/<group>) is false) and fall through to the unchanged main-package logic — all OK.
  • The main-package else branch was diffed against master's test/runtests.jl and is byte-identical (after accounting for the one extra level of else indentation).
  • All 11 modified runtests.jl parse cleanly and are Runic-clean (Runic 1.5.1). No TOML files changed; no test_groups.toml present.

SublibraryCI.yml is unchanged and kept. No bespoke CI_<sublib>.yml is restored. Only Core / QA / GPU group names remain.

Please ignore until reviewed by @ChrisRackauckas.

🤖 Generated with Claude Code

…all groups

Keeps the centralized sublibrary-tests.yml@v1 form (SublibraryCI.yml is
untouched) and fixes the coverage gap by teaching it the real per-sublibrary
test matrix instead of reverting to bespoke CI_*.yml.

Without a lib/<sub>/test/test_groups.toml, the reusable workflow defaults to
GROUP=Core (versions lts/1.11/1/pre) + GROUP=QA (version 1) on ubuntu only, and
emits GROUP="<pkg>" / "<pkg>_QA", which the sublibrary runtests.jl filtered on
bare tag names never match -> most groups ran zero tests, the macOS legs were
dropped, and the per-suite version matrices were lost.

This commit, per sublibrary:

- Adds test/test_groups.toml encoding exactly what the old bespoke CI ran:
  versions 1.10/1.11/1 (lts/1.11/1 for SCCNonlinearSolve), an ubuntu Core
  section, and a parallel <Section>MacOS section with runner="macos-latest" to
  restore the macOS legs. SimpleNonlinearSolve additionally encodes its
  adjoint and alloc_check groups (each on ubuntu + macOS).
- Adds a GROUP-decoding shim at the top of test/runtests.jl that maps the
  emitted GROUP="<pkg>" -> Core and GROUP="<pkg>_<grp>" -> <grp> (stripping the
  "MacOS" runner-only suffix), driving the existing tag/group dispatch. Bare
  "All"/"all" still works for local runs.

Verified with the actual SciML/.github@v1 compute_affected_sublibraries.jl that
each sublibrary emits the same group x version x OS matrix as its old bespoke
CI, and that every emitted GROUP decodes to a non-empty test selection.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Replace the bespoke per-sublibrary group sections (Core/CoreMacOS, and for
SimpleNonlinearSolve adjoint/alloc_check + MacOS variants) with the SciML
standard sublibrary groups:

  Core — all functional/correctness tests (default-on group)
  QA   — Aqua + ExplicitImports + allocation tests (quality/static/allocation)
  GPU  — CUDA tests (driven by the dedicated GPU.yml workflow, GROUP="cuda")

Each affected sublibrary's specialized test/test_groups.toml is deleted so the
reusable SublibraryCI (SciML/.github sublibrary-tests.yml@v1) falls back to its
standard defaults: Core on [lts, 1.11, 1, pre] and QA on [1], all on
ubuntu-latest. None of these sublibraries has a GPU section in test_groups.toml
because CUDA is already handled by the separate GPU.yml workflow.

runtests.jl in each sublibrary keeps the prefix-strip shim
(GROUP="<pkg>" -> Core, GROUP="<pkg>_<Section>" -> Section) and dispatches on the
standard section, folding the old custom groups:
  - functional (core, adjoint, wrappers, basic, EXPANDED/NESTED-style)      -> Core
  - Aqua/JET/ExplicitImports/allocation (alloc_check)                       -> QA
  - cuda/gpu                                                                -> GPU
QA test items are re-tagged from :core to :qa; SimpleNonlinearSolve adjoint
folds :adjoint -> :core and :alloc_check -> :qa; NonlinearSolveSciPy folds
:wrappers and the untagged basic load test -> :core. Also fixes the pre-existing
@itesitem typo in SimpleNonlinearSolve/test/core/allocation_tests.jl so the QA
allocation test actually runs.

macOS legs are dropped: the standard sublibrary CI is ubuntu (Core/QA) plus the
gpu runner (GPU) and does not use macOS. This is a flagged tradeoff — the
previous bespoke groups ran a macOS leg, the standard groups do not.

The centralized SublibraryCI.yml (sublibrary-tests.yml@v1) is kept; no bespoke
CI_*.yml is restored and no workflow files are changed.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude changed the title ci: add sublibrary test_groups.toml so centralized SublibraryCI runs all groups ci: standardize sublibrary test groups to Core/QA(/GPU) Jun 4, 2026
…tcher

Match the OrdinaryDiffEq.jl test/runtests.jl reference exactly. The reusable
SublibraryCI (sublibrary-tests.yml@v1) runs Pkg.test on the root NonlinearSolve
package with GROUP="<sub>" (Core) or "<sub>_<Section>" (e.g. QA). Previously the
root runtests.jl had no sublibrary dispatcher, so sublib tests never ran under
SublibraryCI, and each sublib runtests carried a bespoke prefix-strip shim.

Root test/runtests.jl: add the canonical dispatcher prelude that reads the RAW
GROUP env (before the main-package lowercasing/parsing), detects a lib/<sub>
directory via _detect_sublibrary_group, activates it, transitively Pkg.develop
its [sources] path deps on Julia < 1.11, then runs Pkg.test(<sub>) under a clean
NLS_TEST_GROUP env var. The _detect_sublibrary_group + transitive [sources]
develop + withenv/Pkg.test tail are copied verbatim from OrdinaryDiffEq (env var
adapted to NLS_TEST_GROUP). When GROUP is not a sublibrary (all/downstream/nopre/
bounds/cuda/wrappers/trim), control falls through to the existing main-package
logic, preserved byte-for-byte in the else branch.

Each lib/<sub>/test/runtests.jl: replace the _G/_SUB/startswith prefix-strip shim
with const GROUP = get(ENV, "NLS_TEST_GROUP", "All"); the standard Core/QA(/GPU
via cuda) dispatch is unchanged. Matches OrdinaryDiffEqRosenbrock's plain
get(ENV, "ODEDIFFEQ_TEST_GROUP", ...).

No test_groups.toml (defaults give Core[lts,1.11,1,pre] + QA[1]). SublibraryCI.yml
kept; no bespoke per-sublibrary CI restored.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
@ChrisRackauckas-Claude ChrisRackauckas-Claude changed the title ci: standardize sublibrary test groups to Core/QA(/GPU) ci: align sublibrary CI to OrdinaryDiffEq canonical translating dispatcher Jun 4, 2026
@ChrisRackauckas-Claude

Copy link
Copy Markdown
Member Author

Closing in favor of the project-model approach (per discussion with @ChrisRackauckas).

This PR added OrdinaryDiffEq's GROUP-dispatch _detect_sublibrary_group + transitive [sources]-develop into runtests.jl, but (a) it only touched runtests.jl with no workflow to drive the dispatcher, and (b) it duplicated the [sources]/registry workaround per repo (including ODE-specific comments that do not apply here).

Instead, the [sources] + registry-staleness handling is moving into the reusable workflow (tests.yml, SciML/.github#52), and sublibrary testing will be a matrixed tests.yml@v1 caller with project: lib/X — no per-repo runtests.jl dispatcher, no copy-pasted workaround. A replacement PR will follow once #52 is promoted to v1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants