Skip to content

Keep spelled NonlinearSolution element type concrete across SciMLBase layouts - #1332

Draft
ChrisRackauckas-Claude wants to merge 3 commits into
SciML:masterfrom
ChrisRackauckas-Claude:agent/1619-scc-concrete-solution-type
Draft

ChrisRackauckas-Claude wants to merge 3 commits into
SciML:masterfrom
ChrisRackauckas-Claude:agent/1619-scc-concrete-solution-type

Conversation

@ChrisRackauckas-Claude

@ChrisRackauckas-Claude ChrisRackauckas-Claude commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Please ignore until reviewed by @ChrisRackauckas.

What changed and why

iteratively_build_sols(probs::AbstractVector, ...) previously always typed the sols buffer with the concrete stripped NonlinearSolution (via stripped_solution_type). SciML/SciMLBase.jl#1620 adds a trailing RC type parameter, so that concrete buffer no longer converts into a FunctionWrapper whose signature still spells the ten-parameter SubArray{SSol,...} — SubArray is invariant in its element type.

This is non-breaking: when any vector-form explicitfun is a FunctionWrapper whose second argument is SubArray{SSol,...}, SCC uses that declared SSol as the buffer eltype (legacy ten-parameter spellings keep working, including under #1620 where that spelling is a UnionAll). The lookup keeps an eltype fast path for homogeneous wrapper vectors, then scans every element so a legacy wrapper that is not first (e.g. Any[plain_noop, legacy_FW]) still wins. Otherwise it uses the concrete stripped type so the updated-signature path keeps isconcretetype(eltype(original)). Numerically, step counts and iterates are unchanged.

The updated-signature test derives its SubArray element type from public solve + strip_solution + build_solution (not the internal stripped_solution_type helper).

Fail-before / pass-after

Homogeneous legacy wrappers (round 1)

Negative control on pre-compat src against SciMLBase #1620: SubArray convert MethodError (5 pass, 1 error).

Mixed Any[plain, legacy FW] (round 2, blocking)

Negative control on 98947a2f (eltype/first-only scan) against SciMLBase #1620:

MethodError: Cannot `convert` an object of type
  SubArray{NonlinearSolution{..., SciMLBase.ReturnCode.T}, ...}
to an object of type
  SubArray{NonlinearSolution{...10 params...}, ...}
...
Test Summary:                                                | Error  Total  Time
Mixed Any[plain, legacy FW] negative control (98947a2f scan) |     1      1  6.1s
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.

After scanning every element (40bdc35f), core_tests__item8.jl (incl. mixed case) passes 14/14 against both SciMLBase versions (below).

Verification (Julia 1.12.7)

GROUP=Core julia +1.12 --project=<scratch/scc-head-env> test/runtests.jl   # SciMLBase #1620
GROUP=Core julia +1.12 --project=<scratch/scc-release-env> test/runtests.jl # SciMLBase 3.56.0

SciMLBase #1620:

Manual SCC                                             | 14    14
SCCNonlinearProblem solve without explicit u0 (#758)   |  3     3
SCC Residuals Transfer                                 |  6     6
Vector-form SCCNonlinearProblem with FunctionWrappers  | 13    13
HomotopyProblem SCC block is continued                 |  5     5
AutoDespecialize SCC subproblems                       | 21    21
Vector{Any} SCC container eltype is preserved          | 11    11
Legacy ten-parameter FunctionWrapper SCC signature     | 14    14
HEAD_CORE_EXIT=0

SciMLBase released v3.56.0:

Manual SCC                                             | 14    14
SCCNonlinearProblem solve without explicit u0 (#758)   |  3     3
SCC Residuals Transfer                                 |  6     6
Vector-form SCCNonlinearProblem with FunctionWrappers  | 13    13
HomotopyProblem SCC block is continued                 |  5     5
AutoDespecialize SCC subproblems                       | 21    21
Vector{Any} SCC container eltype is preserved          | 11    11
Legacy ten-parameter FunctionWrapper SCC signature     | 14    14
RELEASE_CORE_EXIT=0

Lint/format:

$ julia +1.12 --project=@runic -e 'using Runic; exit(Runic.main(ARGS))' -- --check \
    lib/SCCNonlinearSolve/src/SCCNonlinearSolve.jl \
    lib/SCCNonlinearSolve/test/core_tests__item4.jl \
    lib/SCCNonlinearSolve/test/core_tests__item8.jl
RUNIC: 0
$ typos <same files>
TYPOS: 0

Not verified

  • Enzyme/other AD extension paths of SCC (Core group only).
  • Wrappers that are not FunctionWrapper (or whose arg types do not expose a SubArray{NonlinearSolution,...}) — those still use the concrete stripped type.
  • Wall-clock performance (type-level selection only).

Reviewer push-back points

  • Detection uses nameof(FW) === :FunctionWrapper plus .parameters rather than a hard dep on FunctionWrappers.jl.
  • When the declared SSol is the ten-parameter UnionAll under #1620, the buffer eltype is intentionally non-concrete so the wrapper keeps converting; concreteness is asserted only in the updated-signature test (core_tests__item4.jl).

Part of SciML/SciMLBase.jl#1619 (companion to SciML/SciMLBase.jl#1620).

Please ignore this draft until reviewed by @ChrisRackauckas.

Cursor Agent CLI 2026.09.26-dd393fe, model auto. Transcript: /log.txt

Adopted 2026-10-07 (ci-green head)

What changed: rebased onto master 0055e314 (after #1321, #1356 and #1357). The rebase had no conflicts and made no code change. Head is now 31046598. All original commits are kept.

Where the fix belongs: SciMLBase v3.57.0 (SciML/SciMLBase.jl#1620, released) added the trailing RC parameter on purpose. Julia has no default type parameters, so no constructor or alias in SciMLBase can make the ten-parameter spelling NonlinearSolution{T,…,Tr} concrete again. The fix has to come from SCC choosing a concrete buffer eltype, which is what this PR does. The SciMLBase PR body itself names this PR as its companion. ModelingToolkit's SCC codegen (src/problems/sccnonlinearproblem.jl) does not spell the solution type. The legacy FunctionWrapper path therefore serves user code that spells the ten-parameter type, such as the old core_tests__item4.jl.

Verification: GROUP=Core Pkg.test() in lib/SCCNonlinearSolve, SciMLBase 3.57.0, on Julia 1.10.12 and 1.12.7.

Master 0055e314 fails on both versions:

Vector-form SCCNonlinearProblem with FunctionWrappers: Test Failed at .../lib/SCCNonlinearSolve/test/core_tests__item4.jl:100
  Expression: isconcretetype(eltype(scc_sol_debug.original))

This branch passes. Testing SCCNonlinearSolve tests passed on both versions:

Vector-form SCCNonlinearProblem with FunctionWrappers |   12/13  (lts/1)
HomotopyProblem SCC block is continued (autodiff threaded) | 7 7
AutoDespecialize SCC subproblems                      |   21    21
Vector{Any} SCC container eltype is preserved         |   11    11
Legacy ten-parameter FunctionWrapper SCC signature    |   13/14  (lts/1)

(The lts and 1 counts differ by one because of the VERSION >= v"1.11" compile-time assertion.) Runic --check and typos on the changed files: clean.

Red checks after the rebase (CI is still running):

Risk assessment

  • Risk: low
  • Blast radius: SCCNonlinearSolve's vector-form iteratively_build_sols buffer eltype selection, and only that. Tuple-form SCC is untouched, and no public API changes. The FunctionWrapper detection is structural: it checks nameof and the SubArray parameters, with no FunctionWrappers dependency. An opaque wrapper falls back to the concrete stripped type.
  • Evidence: the SCC Core group fails on master and passes on this branch on lts and Julia 1 (above).
  • Independent review: pending
  • Merge: auto-merge candidate once SCC Core is green on CI. A reviewer may push back on the reflection-based legacy-wrapper detection. It exists to keep spelled ten-parameter FunctionWrapper signatures converting, and dropping it would break those users.

🤖 Generated with Claude Code (model: claude-opus-5-5[1m]) https://claude.ai/code/session_011aK5Xvp9NpdkzKuEwtiMGd

Independent review (added by the fleet master)

Verdict: MERGE (reviewed head 3104659)

  • Risk: medium
  • Blast radius: Vector-form SCCNonlinearProblem solution buffers in SCCNonlinearSolve.iteratively_build_sols. The ten-parameter NonlinearSolution{T,N,uType,R,P,A,O,uType2,S,Tr} spelling is a UnionAll on current SciMLBase (the eleventh parameter is the retcode carrier). stripped_solution_type applies SciMLBase.ReturnCode.T when that spelling is not concrete. If an explicitfun FunctionWrapper declares a SubArray element type, the buffer uses that declared type so the invariant SubArray still converts. Tuple-form SCC is unchanged. No new dependency. SCCNonlinearSolve stays at 1.16.0. A concrete solution is still an instance of the old ten-parameter spelling, so this is a tighter element type, not a new solution struct.
  • Evidence: julia +release 1.13.2 with SciMLBase v3.58.0: the ten-parameter spelling is not concrete; applying ReturnCode.T is concrete. On master dd8a550b79c4c9ea72a7b95b9233a34ffdb5bd8e, core_tests__item4.jl failed one assertion and passed 12: isconcretetype(eltype(scc_sol_debug.original)) was false for NonlinearSolution{Float64,1,Vector{Float64},Vector{Float64},@NamedTuple{p::Nothing},Nothing,Nothing,Nothing,Nothing,Nothing}. With head 310465988d89acb6abee6f51bdae509cede8dc75 sources, Pkg.test of SCCNonlinearSolve passed every core file: item4 13, item8 (the new legacy ten-parameter wrapper, including Any[plain, legacy]) 14, and items 1–3 and 5–7 (14, 3, 6, 7, 21, 11). CI sublibrary Core on Julia 1, lts, and pre, plus QA and the SCC downgrade job, passed: https://github.com/SciML/NonlinearSolve.jl/actions/runs/37638822372. benchmark (lts) failed while resolving, before any benchmark ran: NonlinearSolveFirstOrder 2.10.0 requires SciMLJacobianOperators 0.1.20 and that run's registry only offered 0.1.0–0.1.19 (https://github.com/SciML/NonlinearSolve.jl/actions/runs/37638820774/job/112852033721). benchmark (1) passed. The three-dot diff does not touch FirstOrder or JacobianOperators. Compare vs master: 3 ahead, 3 behind (Bump PETSc in / #1329, Bump NonlinearSolveBase and SCCNonlinearSolve for release #1371, Allow JET 0.12 in the QA environment for Julia 1.13 #1373), MERGEABLE.
  • Independent review: grok-cli grok-4.7 rated it medium, high: master stores a non-concrete ten-parameter element type, the head's core suite passes, and the legacy wrapper path keeps the caller's declared element type so SubArray conversion still works
  • Merge: needs human review — the element type of a public SCC solution vector changes, and the red benchmark (lts) job is a registry resolve rather than a result of this diff

🤖 Section added by an AI agent — harness: Claude Code 2.1.300 · model: claude-opus-5-5 (review by grok-cli, grok-4.7)
Conversation: https://claude.ai/code/session_01VZjwf7Ux9YtTzwHGHMqnsq

🤖 Automated comment from an AI agent running as @ChrisRackauckas — not written or reviewed by Chris.

Risk assessment (independent review, Codex gpt-6.1-sol, head 3104659)

  • Verdict: CHANGES

  • Rating: Codex gpt-6.1-sol rated it high (H1) at head 3104659

  • Risk: high (H1: public vector-form SCC solves reject previously working callback and return-code-carrier inputs; no major version bump accompanies these regressions).

  • Blast radius: Vector-form SCCNonlinearProblem, including ordinary callable adapters around legacy FunctionWrappers and algorithms returning convertible return-code carriers. Tuple-form SCC and the default Newton/trust-region/polyalgorithm lanes are untouched. The change affects compatibility with SciMLBase ≥3.57, rather than numerical iteration formulas.

  • Evidence: Read the linked issue and companion PR, the complete git diff origin/HEAD...HEAD, the surrounding SCC implementation, tests, and CI results. git diff --check origin/HEAD...HEAD passed. Ran Julia 1.12.7 with released SciMLBase 3.58.0, four threads, the requested depot, and one test process at a time. Sources were archived into scratch; only the SCC source path differed between comparisons, with shared dependencies held identical. Default SCC source: origin/HEAD at 8d5e163e724ddfc1232055175dd8fb592915809f. The earlier companion-source comparison gave the same intended-fix/carrier results.

    Commands (where RUN=<local path>, JULIA=<local path>):

    export JULIA_DEPOT_PATH="$RUN/depot:$HOME/.julia" JULIA_NUM_THREADS=4
    export JULIA_NUM_PRECOMPILE_TASKS=1 JULIA_PKG_PRECOMPILE_AUTO=0 TMPDIR="$RUN/scratch/tmp"
    for revision in head base; do
      timeout 3600 "$JULIA" --startup-file=no --compiled-modules=existing \
        --project="$RUN/scratch/$revision-release-env" \
        "$RUN/scratch/targeted-release.jl" "$RUN/scratch/head"
    done

    Observed results:

    Target Default SCC source PR head
    Updated core_tests__item4.jl 5 pass, 1 conversion error 13 pass
    New core_tests__item8.jl, unchanged assertions 14 pass 14 pass
    Closure forwarding to legacy wrapper 2 pass 1 conversion error
    Custom return-code carrier 3 pass 1 typeassert error

    Both aggregate runs exit 1 for the distinct errors above; neither is represented as a passing suite. Logs: head, default. Reproducer: targeted runner, carrier.

    G1: not applicable — no numerical algorithm, tolerance, or initialization formula changes; the SCC tests compare against a separately assembled monolithic problem. G2: not applicable to the edited code — no GPU kernels, AD rules, Reactant tracing implementation, or MPI code is changed; actual end-to-end Reactant/AD compatibility remains unverified, and ordinary carrier compatibility already fails. G3: not applicable — the allocated buffer is local to a solve; no shared cache, threading, or global default is changed.

    The red benchmark (lts) check is demonstrably a default-branch failure: its log first selects BASE_SHA=0055e314e78fcd070007ecaaeb6702116a279631, announces benchmarking that revision, then errors resolving SciMLJacobianOperators 0.1.20 while the registry offers only through 0.1.19. It never reaches benchmarking the PR revision. This is evidence from the actual base execution, not an assumption that master also fails.

  • Merge: needs changes (fix both compatibility regressions and add discriminating tests). If rejecting those inputs is intentional, treat it as a breaking change requiring a release decision rather than calling this non-breaking.


🤖 Posted by an AI agent — harness: Codex CLI 0.162.1 · model: gpt-6.1-sol
Conversation: local headless review run (no public session URL); transcript kept by the fleet master.

ChrisRackauckas and others added 3 commits October 7, 2026 10:03
…Base layouts

`iteratively_build_sols(probs::AbstractVector)` spells out
`NonlinearSolution{...}` to type the `sols` buffer, and the vector-form
test spells the same type into the `FunctionWrapper` `SubArray` signature.
SciMLBase #1620 adds a trailing `RC` type parameter to `NonlinearSolution`,
which turns that application into a UnionAll: `Vector{ST}` then has an
abstract eltype and `isconcretetype(eltype(original))` fails.

`stripped_solution_type` spells the ten non-`RC` parameters and applies
`ReturnCode.T` only when the spelled result is not already concrete, so
`ST` is the concrete stripped-solution type under both the released
SciMLBase layout and the #1620 layout. Verified: GROUP=Core passes
against SciMLBase 3.56.0 and against the #1620 branch, including the
vector-form concreteness assertion.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Devin <noreply@cognition.ai>
Agent-Harness: Devin CLI 3000.11.3
Agent-Model: swe-2-high
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/SciMLBase.jl/jobs/1619-devin/log.txt on amdci2.julia.csail.mit.edu
Vector-form SCC users wrap explicitfuns with a FunctionWrapper whose
second argument spells SubArray{SSol,...}. Prefer that declared SSol
over the concrete stripped_solution_type so legacy ten-parameter
signatures keep converting under SciMLBase #1620's trailing RC param.
Add a regression test for the old signature next to the updated one.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/NonlinearSolve.jl/jobs/1332-cursor/log.txt on amdci2.julia.csail.mit.edu
sols_buffer_eltype previously only inspected eltype/first, so
Any[plain_noop, legacy_FW] still built a concrete buffer and broke
conversion. Scan every element after the eltype fast path. Derive
item4's SubArray element type from public strip_solution instead of
the internal stripped_solution_type helper, and cover the mixed
container in item8.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Cursor Agent <noreply@cursor.com>
Agent-Harness: Cursor Agent CLI 2026.09.26-dd393fe
Agent-Model: auto
Agent-Session: local session, transcript at /home/crackauc/sandbox/agent-jobs/NonlinearSolve.jl/jobs/1332-cursor/log.txt on amdci2.julia.csail.mit.edu
@ChrisRackauckas-Claude
ChrisRackauckas-Claude force-pushed the agent/1619-scc-concrete-solution-type branch from 40bdc35 to 3104659 Compare October 7, 2026 14:41
ChrisRackauckas-Claude pushed a commit to ChrisRackauckas-Claude/NonlinearSolve.jl that referenced this pull request Oct 21, 2026
…, reinit!(u0,p)

Review (Codex, gpt-6.1-sol) on the init/step!/solve! cache found four correctness
issues, all confirmed by running probe scripts:

1. A second solve! on an already-completed cache recombined an empty history
   (the fast "no stepping yet" path advanced idx past the end without
   retaining any block solutions), returning a spuriously successful empty
   u = Any[]. The fast path now builds the per-block solutions via
   iteratively_build_sols (the same driver _scc_solve itself uses) and
   retains them on the cache, so any later solve! - including after a
   completion no-op step! - recombines the real result instead.

2. step! always passed Vector{Any} to explicitfuns!, breaking
   tuple-specialized callbacks (MethodError) and the FunctionWrappers vector
   path solve itself supports (conversion error). step! now passes the exact
   representation the existing driver passes: a growing Tuple for tuple-form
   blocks, a typed view into a concretely-typed, growing Vector for
   vector-form blocks.

3. With store_original=Val(true) on vector-form blocks, stepping normalized
   everything into a Tuple, so typeof(direct) != typeof(stepped). Fixed by
   (2): cache.sols is now exactly the container _scc_solve/iteratively_build_sols
   produce, so the shared __scc_combine_sols helper (extracted from
   _scc_solve's tail, used by both the fast and stepped paths) always agrees
   on original's container and element types.

   The concrete per-block solution type for vector-form blocks is derived via
   a new shared helper, __scc_vector_sol_type, factored out of the existing
   vector-form iteratively_build_sols method rather than hand-spelled again -
   deliberately, since the hand-spelled 10-parameter NonlinearSolution
   signature is what core_tests__item4.jl's pre-existing failure traces to
   (SciMLBase#1620 added an 11th type parameter; SciML#1332
   is the fix for that, in progress separately). Sharing the helper means
   this cache's types track the existing driver's, including through
   whichever of SciML#1332 or a SciMLBase release lands first.

4. reinit! accepted no replacement u0/p. Added
   reinit!(cache, u0 = missing; p = missing), delegating entirely to
   SciMLBase.remake(cache.prob; u0, p) (which already knows how to chunk u0
   across blocks and how parameters_alias governs distributing p) rather than
   reimplementing that logic, then resetting progress on the result.

Also (non-blocking in review): declared SCCNonlinearCache public via
SciMLPublic.@public (the Julia 1.10-compatible mechanism SteadyStateDiffEq.jl
already uses for its analogous cache types), and bumped the minor version
(1.16.0 -> 1.17.0) for the new public API.

All four fixes verified against the reviewer's own probe scripts
(probes.jl, reinit_public.jl) before writing the corresponding assertions
into core_tests__item8.jl. Pre-existing tests (item1, item2, item3, item5,
item6, item7) rerun directly and still pass unchanged.

Co-Authored-By: Chris Rackauckas <accounts@chrisrackauckas.com>
Co-Authored-By: Claude Sonnet <noreply@anthropic.com>
Agent-Harness: Claude Code 2.1.251 (subagent)
Agent-Model: claude-sonnet
Agent-Session: https://claude.ai/code/session_01NB3oFTNzGW79UtDt8AyjsM

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants