chore(vetting): one coverage-scanner library, and a feature report that checks the registry against the tree

[darkclad]

30 files, 29 under tests/vetting/ and one CI workflow. No src/nyx change —
git rev-parse HEAD:src equals git rev-parse upstream/main:src. Net −1,133 lines.

Four commits, all chore(vetting), so nothing here reaches the CHANGELOG and nothing moves the
version whether this is squashed or merge-committed. That is deliberate: release-please-config.json
sends feat to Features and bumps the patch pre-1.0, and refactor to Code Refactoring, and none of
this is observable to anyone using Nyxus.

---

The problem

oracle_coverage.csv records what a row claims — a status, an oracle, a tolerance, a recipe.
The per-family scanners under audit/ record what the tree asserts — which gtest case covers
which feature, under which oracle, with which drift guard behind it.

Neither could check the other. coverage_report.md is a sixteen-row tally of the registry against
itself; it cannot see a test at all. The two sides were only ever compared per family, by hand, when
somebody happened to run a scanner — and five families had no scanner to run.

That is the gap this closes, and it needed the three steps in order.

---

1. The one family with no coverage artifact

3D GLDZM was the only family in the registry with no coverage artifact at all: scan_gldzm_coverage.py
is 2D-only, so its 18 rows had nothing to be checked against, scanned or hand-written. Five other
family x dim combinations do have an artifact, but nothing re-derives it from the tree -- the report
names all five under Families with no scanner, and writing those scanners is separate work.

A gldzm_3d_mirp_vetting_report.md already existed, so the vetting had been done; only the artifact
that re-derives it from the tree was missing.

scan_gldzm3d_coverage.py + gldzm_3d_coverage.csv close it, modelled on the 3D NGLDM scanner —
the other family with no oracle test, for the same reason. ORACLE_SUFFIX is deliberately empty:
MIRP is config-matched and reproducible, but disagrees with Nyxus on all 16 features it computes by
up to 67.5×, and an independent from-definition recomputation (26-connected zones, city-block
distance transform) reproduces MIRP to rel 3.2e-16. The definition is reachable and Nyxus is not
computing it, so nothing here may be promoted on the current implementation.

Its --check runs the assertion-identity tier, which is the tier that was missing when 3GLDZM_ZDM
carried a golden of 222 — a factor of 14.5 off — with no test function and no TEST()
registration, so nothing could fail on it.

gldzm_3d_golden_regen.md records the regeneration path and one gap it does not close: this is the
only 3D family with no *_dump_regression test, so its pin table is transcribed by hand. That is a
C++ change and needs the ASan gate; it is not in this PR.

2. One library instead of twenty copies

not_covered.md already recorded that the coverage logic is copied across every scan_* file. The
cost was measurable: the twenty had drifted into four spellings of the function regex and two of the
comment stripper, and a fix applied to one copy stayed in one copy.

audit/scanlib.py holds it once — the coverage rule, the scan, the acceptance checks, the artifact
rendering and the run loop. 4,993 lines across 19 scanners become 1,715 across 20, plus a 550-line
library. Thirteen families are now a docstring and one Family(...) call, the shortest 33 lines.

What is deliberately not unified, because the difference is real and belongs where the family is
declared:

family	what it does differently
2D NGTDM, IMQ, 3D GLDM, 3D GLSZM	read current_test per test kind — a vetted row answers to the oracle files, a regression row to the snapshot one. Stricter than the shared per-feature reading, and catches a class it cannot
2D moments	resolves coverage through golden tables; nothing in it names a feature on an assertion line
2D radial	declares test_2d_radial_mechanics.h UNCREDITED — its pins characterize known defects, so crediting them would make the defects acceptance criteria

Each keeps its own collect or checks and says why in its own docstring.

Three behaviour changes, all in 2D morphology, all corrections

a. helper_features over-credited coverage in 11 of 14 scanners. A module-level pytest helper's
body was bounded at the next helper rather than the next top-level def, so the last helper in a
file swallowed every test function below it and each caller inherited every feature name in the
file. Three scanners carried the fix; eleven did not. scanlib takes the correct bound, which exposes
eight current_test cells naming a file that covers nothing for them:

• four chord rows (MAXCHORDS_MAX/MIN, ALLCHORDS_MAX/MIN) named test_2d_morphology_invariant.py,
  which reads those four in an if guard and asserts only the _ANG features;
• four PERIMETER rows named test_2d_ooc_invariant.py — whose own docstring says so:
  "PERIMETER is deliberately NOT among them: it does NOT agree, and that divergence is pinned as a
  regression." The rows and the file contradicted each other outright and nothing could see it.

Each of the eight keeps its other files; none is left empty.

b. A pytest case can assert features that rule cannot see. test_2d_ooc_invariant.py names the
six ellipse features in a local tuple and EROSIONS_2_VANISH in the comprehension that builds its
compared-column list, then loops that list while asserting — so no name reaches an assertion line.
Under the corrected bound those seven would have read as uncovered, which would have been wrong:
the case really does compare them. scanlib gains py_loop_tables, the Python twin of the
loop_tables concession the C++ equivalence tests already had, gated the same way — the function
must both assert and range-loop, so a plain lookup list is not coverage. Opt-in; 2D morphology is the
only family that takes it.

c. morphology_2d_coverage.csv repeated eight features four times each — 123 lines for 99
features, byte-identical repeats carrying no extra information. That scanner rendered one line per
registry row, and the registry has moved to one row per config cell. The artifact is a
feature→test rollup, so scanlib renders one line per feature and the 24 duplicates are gone. Any
family whose registry grows config-cell rows would have hit this.

3. The report, and the CI gate

report_features.py generates features.csv (923 rows) and features.md, one row per
(dim, feature, oracle, config_recipe) — the key the registry is already unique on, and the one
that keeps a feature's config cells apart, which matters because a feature vetted at ibsi=true can
diverge by two orders of magnitude at ibsi=false. --check enforces that uniqueness; it was true
but unchecked, which is the shape of #443's C3.

Columns are grouped claim_* / scan_* so a reader can see which side of the join a value came
from, and the derived verdict says whether they agree. That is what makes this a gate rather
than a summary.

It reads the tree through the scanners themselves, importing each scan_*_coverage.py and
running its FAMILY through scanlib, so the report and the per-family --check cannot disagree
about what a test covers. That is only possible because step 2 gave them a shared library; without
it, a tree-wide report means a twenty-first copy of the coverage rule — which is exactly how
report_feature_tests.py came to credit tests with features they never checked.

What it found

745 of 923 rows agree. None disagree. 178 (19%) come back unscanned:

dim	family	rows	features
2D	firstorder	36	36
2D	glcm	59	59
2D	glrlm	32	32
2D	gabor	2	1
3D	firstorder	49	36

These five have an audit/<family>_<dim>_coverage.csv with no scanner behind it. Those artifacts
record what someone believed rather than what the tree asserts, and no check anywhere could say so —
including all of first-order and all of 2D GLCM, two of the most-claimed families. They are declared
in KNOWN_UNSCANNED with a reason each, features.md gives the number a section of its own, and
--check fails if a sixth family appears undeclared. Writing those five scanners is the next branch;
2D GLCM and 2D GLRLM each have a working 3D twin to copy.

CI gains two steps beside the existing two: the twenty scanners in one loop, and
report_features.py --check. Both are stdlib-only and need no build.

4. What review caught

A fourth commit, after review of the three above. Two of the four are the class of defect this PR
exists to prevent — a check that goes quiet, and a generated artifact that says something untrue —
so they are worth naming rather than folding into the others.

A dropped acceptance check, in two families. Three pre-refactor scanners ran
no_oracle_claimed, the reverse of oracle_mismatch: an oracle-suffixed test asserts a feature
while the row claims no oracle at all. Moving to declarations carried it forward for 2D radial and
silently dropped it for 2D neighbor and 2D zernike — the other two families holding rows that
claim no oracle, which is the only place it can fire. Restored in both declarations. The
regression is reproducible: blank the oracle cell on those nine rows and the pre-refactor
scanners report 8 and 1 problems where the post-refactor ones reported clean; they now report 8
and 1 again. Nothing else covered it — report_features.py's verdict for a vetted row with an
empty oracle cell is agree, because the empty cell short-circuits the mismatch branch.
scanlib.py had also recorded the loss as if it were the inherited state ("Only 2D radial runs it
today"); it now names all three.

94 rows pointed at the wrong dimension's config matrix. pointers() fell back from
matrix/<family>3d.md to matrix/<family>.md for a 3D family, and three 3D families have no 3D
matrix — so 3D GLCM (59 rows), 3D GLDZM (18) and 3D morphology (17) each linked a file documenting
the 2D calculator, matrix/morphology.md being titled "2D morphology config matrix" outright.
The fallback is gone; an empty cell is the honest reading of no 3D matrix exists yet, and it is
what every other missing pointer already renders.

235 rows could not reach their own vetting report. The vetting_report pointer was keyed on the
row's claimed oracle, so a family whose oracle disagrees with Nyxus — which claims none on purpose
— could never link the report that recorded the disagreement. 3D GLDZM was the clearest case: this
PR adds its scanner, whose ORACLE_SUFFIX={} is justified entirely by
gldzm_3d_mirp_vetting_report.md, and all 18 of its rows rendered an empty cell. The pointer now
takes the claimed oracle first, then the oracles the tree asserts against, then — only when neither
side names one and the family x dim has exactly one report — that report. Rows with a
vetting_report go from 637 to 872.

And four smaller ones. The CI scanner loop exited on the first failing family, so a change
breaking the shared library reported one family instead of twenty; it now runs all twenty and fails
once. The family x oracle matrix in features.md keyed on family alone, merging 2D and 3D for the
nine family names that exist in both — 2D morphology's matlab 33 and 3D morphology's mirp 8 sat
in one row; it is keyed by dim now, and every column total is unchanged, which is the invariant that
says the old table mislabelled rather than miscounted. scan_gldm3d's write line had lost its
feature count. The README described audit/features.csv, a git-ignored byproduct of the retired
pipeline, two sections below the table linking the new tracked features.csv; the two share a
basename and nothing else, and the README now says so.

No audit/*_coverage.csv changed, and outside the three pointer columns no features.csv value
changed — row identity and order included.

---

New artifacts

Path	What
tests/vetting/audit/scanlib.py	The shared scanner library
tests/vetting/audit/scan_gldzm3d_coverage.py, gldzm_3d_coverage.csv	The missing family
tests/vetting/audit/gldzm_3d_golden_regen.md	Its regeneration guide
tests/vetting/report_features.py	The report generator and its --check
tests/vetting/features.csv, features.md	The report

How it was verified

Not "the tests pass" — the scanners' behaviour was pinned and compared.

• A behavioural baseline, re-captured on this base. Each scanner's artifact and its --check
  output were recorded under a four-mutation battery applied to the registry: blank a test_name,
  force a row to vetted, point current_test at a missing file, corrupt an oracle token. Byte-
  identical CSVs alone would not catch a weakened acceptance check. After the rebase onto 5aca80b9
  the baseline was re-captured from the pristine upstream scanners on the new base rather than
  carried over. All twenty match except 2D morphology's three intended differences above.
• Negative controls on every new assertion shape. The gldzm-3D scanner reports the fault when a
  case is unregistered, when a feature is dropped from an assertion line, and when a row names a
  bogus test_name. py_loop_tables is controlled in both directions — emptying the ellipse tuple
  drops exactly those six features, and a list the function never range-loops is not credited. The
  report's key gate fires on a planted duplicate row and its unscanned gate on a deleted scanner.
• The dropped check was pinned by its own reproduction, not by reading. Blanking the oracle
  cell on the nine 2D neighbor and 2D zernike rows makes the restored check report 8 and 1 problems
  — the counts the pre-refactor scanners produce on the same injection, and the counts the
  post-refactor scanners reported as clean before the fix.
• The pointer fixes were diffed against the previous features.csv row by row. The only cells
  that went from set to empty are the 94 wrong-dimension matrix links; 235 gained a
  vetting_report; no non-pointer column changed and row identity and order are unchanged. Each of
  the six family x dim groups the single-report fallback fires on was checked against the report it
  links — including the two whose registry spelling differs from the report's
  (gldzm_3d_mirp_vetting_report.md tabulates all 16 as LDHGLE, LDE, … without the 3GLDZM_
  prefix), which an exact-name search reads as a miss.
• 20/20 scanners --check clean; check_coverage.py --check clean and --write reproduces
  coverage_report.md unchanged; check_test_names.py --check 0 violations;
  report_features.py --check clean (923 rows).
• pytest tests/python/ — 98 passed, 1 skipped, 7 failed. The 7 are pre-existing HU and
  signed-int16 failures that reproduce identically on an untouched worktree with the same extension;
  no C++ and no python test is touched by this branch.
• ASan is not applicable and was not run: src/ is byte-identical to upstream/main.

Left open, deliberately

• scan_intensity_histogram_coverage.py counts a bare readout as coverage (fvals[(int)...]
  lines as well as assertion lines). Every other scanner refuses to, for the reason the shared
  docstring gives. Preserved so this refactor changes no behaviour, and isolated in that family's
  collect_override where it is visible. Removing it will drop coverage the family currently claims,
  so the affected rows need re-examining one by one — its own pass.
• The five unscanned families, as above.
• 3D GLDZM's missing dump test, as above.

[vjaganat90]

Overall looks good but these need a tweak.

[vjaganat90]

[Blocker] The verdict is not actually config- or assertion-specific. The report declares (dim, feature, oracle, config_recipe) as its key because configurations can disagree by orders of magnitude, but verdict_of() only consults feature-wide oracle/regression buckets. It never uses config_recipe, claim_test_name, or the row's claimed file to select the corresponding assertion. I reproduced this with the vetted 3GLSZM_GLN row: changing its recipe from glszm3d.pyradiomics_bincount20 to either glszm3d.regression_constant_roi or glszm3d.regression_ut_phantom_nobinning still returns agree. The generated rows likewise receive the same complete list of oracle and regression tests regardless of which configuration they claim. Please make the scanned evidence row-scoped—ideally by resolving a general assertion identifier to its function and configuration—or narrow the report's contract to feature-level coverage. As written, the central config-aware guarantee is not enforced.

[darkclad]

Fixed — the verdict is assertion-specific where the row identifies an assertion, config-specific where the family can say what a config looks like, and the rest now says so per row instead of implying otherwise.

• test_name resolves through test_all.cc to the function it runs, and the verdict asks whether that function is the row's evidence. 307 rows carry one; all 307 resolve and agree, so this is a strengthening the tree already satisfied.
• Family.recipe_reader maps a recipe to the pattern its asserting function's name must match. 3D GLDM had one; 3D GLSZM and 3D NGTDM gain one — the three families whose recipes are distinguishable by function name rather than by file or kind. 151 rows are config-checked, and your reproduction (3GLSZM_GLN's recipe swapped for glszm3d.regression_constant_roi) now returns recipe-mismatch.
• A new verdict_scope column reads row+config (151) / row (156) / feature (438) / none (178). The 438 carry no test_name at all, so nothing identifies the assertion they describe — the honest reading is feature-level, and the column says that rather than letting the key imply more. PR/todo.md item 58 tracks closing them per family.
• scan_row_tests shows the functions the row's own test_name resolved to.

Controls are in audit/scanlib_selftest.py (runs in CI): swapped recipe, undeclared recipe, test_name pointing at another feature's case, test_name naming no case, and a blanked test_name falling back to feature.

[vjaganat90]

[Blocker] Helper propagation still converts unasserted values into coverage. Every feature read by a helper is credited to every test that calls that helper, even when the caller discards part of the result. There is a current example in test_2d_morphology_fraclac.py: _fd() returns both FRACT_DIM_BOXCOUNT and FRACT_DIM_PERIMETER; the box-count test does bc, _ = _fd(...), while the perimeter tests do _, pf = _fd(...). Nevertheless, the generated artifact credits all three tests with both features. That contradicts the stated rule that coverage requires an assertion. Please make helper attribution aware of which returned value reaches the assertion, or provide an explicit family-level mapping for multi-value helpers. A negative test for “helper returns two features, caller asserts only one” would pin the intended behavior.

[darkclad]

Fixed — helper attribution is now per returned position, and a caller is credited only with the positions it both binds and asserts on.

helper_positions() maps each element of a helper's tuple return to the features that reach it, following the local the column lookup named (bc = [c for c in row.index if c.endswith("FRACT_DIM_BOXCOUNT")][0], then return float(row[bc]), float(row[pf])). helper_credit() reads the caller's unpacking against that: _ is dropped, and so is a bound name that never reaches an assertion line. A caller that does not unpack keeps the whole helper, as does any feature the return positions do not account for.

Your example is the measured result — morphology_2d_coverage.csv changes in exactly two cells:

• FRACT_DIM_BOXCOUNT loses test_2d_morphology_perimeter_disk_analytic and ..._koch_snowflake_analytic
• FRACT_DIM_PERIMETER loses test_2d_morphology_boxcount_known_dimension_analytic

Both keep the gtest case test_2d_morphology_fractal_dimension_blob512_fraclac, which asserts both on its own assertion lines and does not go through _fd.

The negative test you asked for is in audit/scanlib_selftest.py: a helper returning two features with one caller per half, a caller that does not unpack (keeps both), a single-value helper (unchanged), and a caller that binds the second value but never asserts on it (still not credited).

[vjaganat90]

[Blocker] py_loop_tables credits every local literal when any loop exists. This check only establishes that the function contains some for … in <name> loop; the code then extracts features from every local list or tuple literal without checking whether that literal feeds the loop or its assertion. For example, a function containing unused = ("PERIMETER",), checked = ("AREA",), and a loop over checked reports both features as covered—I reproduced that directly through py_literal_features(). This reintroduces the false-positive coverage this refactor is meant to remove. Please associate each assignment's left-hand name with the collection that is actually iterated—following the current ellipse → cols flow where needed—and add an unused-literal-beside-a-real-loop negative control.

[darkclad]

Fixed — literals are resolved by name against the loop, so a literal the loop never reaches is not credited.

PY_ASSIGN_LITERAL now captures the assigned name and PY_LOOP_NAME the iterated one; only the iterated literal is credited, plus any literal a reached literal names. That second hop is the ellipse → cols flow you pointed at: cols = [c for c in df_ram.columns if ... or c in ellipse], then for c in cols:.

Your exact case is a control in audit/scanlib_selftest.py — unused = ("PERIMETER",) beside checked = ("AREA",) and a loop over checked credits AREA only. The seven features the concession exists for in test_2d_ooc_invariant.py are credited exactly as before, so no artifact moves.

Your follow-up (loop not tied to an assertion) is answered on the round-2 comment below and is fixed too.

[vjaganat90]

[Blocker] The row check still accepts the wrong evidence kind and oracle. For every non-vetted row, covering unions regression, invariant/other, and oracle functions, so a regression claim can be satisfied by an oracle case. I reproduced this by changing the NGTDM_COARSENESS regression row's test_name to TEST_NYXUS.TEST_2D_NGTDM_COARSENESS_IBSI; it still returns ('agree', 'row'). The vetted analogue also passes: the same feature's oracle=mirp row still agrees when test_name names only the IBSI case, because the oracle check above is feature-wide and is not tied to the resolved function. Once a row supplies test_name, please require its resolved evidence to match the exact claim—claimed oracle for vetted, regression bucket for regression, and the corresponding other bucket for invariant—and add both mutations as negative controls.

[darkclad]

Fixed — a row's evidence is now read at its own kind and its own oracle, and the two ways of missing are told apart.

row_covering() returns the row's own bucket rather than a union: the oracle set filtered to the tool the row claims for vetted, the regression set for regression, the invariant set otherwise. Both of your mutations fail now:

• 2D NGTDM_COARSENESS regression row → TEST_2D_NGTDM_COARSENESS_IBSI gives row-test-wrong-kind
• the same feature's oracle=mirp row naming only the IBSI case gives row-test-wrong-oracle (naming the MIRP case still agrees)

Filtering by the claimed oracle needs a function → oracle map, which is scanlib.oracle_token: the SPEC 2 name suffix, plus a Family.fn_oracle exception for a function whose suffix names no oracle. I measured that rather than assuming it — test_3d_gldm_compatibility (3D GLDM's PyRadiomics case through the Python API) is the only asserted function in the tree in that position, and the map it already used privately is declared on the Family now so the report reads the same one.

Both mutations are pinned in audit/scanlib_selftest.py, each beside the unmutated row. No row in the registry fails the stricter rule: 745 still agree and no artifact changes.

[vjaganat90]

[Blocker] Feature and recipe can be proven by different functions. The feature check and recipe check are separate existential tests over fns, so two individually wrong cases can combine into row+config. On the 3GLSZM_GLN pyradiomics_bincount20 row, I set test_name to TEST_NYXUS.TEST_3D_GLSZM_IBSI_GAPPED_PYRADIOMICS;TEST_NYXUS.TEST_3D_GLSZM_SAE_PYRADIOMICS: the first asserts GLN at the wrong recipe, the second matches the requested recipe but asserts SAE, yet the verdict is still ('agree', 'row+config'). Please require one resolved function to be both in the row's exact covering set and matched by the recipe reader, and pin this split-evidence case in the self-test.

[darkclad]

Fixed — one resolved function has to answer the whole row.

The two tests are no longer independent: the recipe reader now ranges only over hits, the functions already established as this row's evidence by row_covering(), so a case that matches the recipe but asserts a different feature can no longer supply the half the other case is missing.

Your split-evidence case is the control, in audit/scanlib_selftest.py: 3GLSZM_GLN's pyradiomics_bincount20 row with test_name set to TEST_3D_GLSZM_IBSI_GAPPED_PYRADIOMICS;TEST_3D_GLSZM_SAE_PYRADIOMICS now returns ('recipe-mismatch', 'row+config') instead of agree.

[vjaganat90]

[Blocker] py_loop_tables now ties a literal to a loop, but still does not tie that loop to an assertion. scan() only establishes that the function contains some assertion somewhere. For example, checked = ("AREA",) followed by for c in checked: print(c) and then an unrelated assert 1 == 1 still reports AREA as covered. That conflicts with the documented rule that the function loops the table while asserting and still lets a lookup/logging loop become coverage. Please require the relevant loop body to contain the assertion (and ideally use its iterator), then add a loop-plus-unrelated-assert negative control.

[darkclad]

Fixed — the loop body has to do the comparing, not merely exist.

A for x in NAME: credits NAME only if its body either asserts, or accumulates into a name the function later asserts on. Your example — for c in checked: print(c) beside an unrelated assert 1 == 1 — credits nothing now.

One note on why it is not "the assertion must be inside the loop body": the case this concession exists for does not do that. test_2d_ooc_invariant.py collects into bad inside the loop and asserts not bad outside it, which is the ordinary accumulate-then-assert shape. Requiring the assertion inside the body would have dropped the seven features the rule was written to credit, so the rule follows the value instead: the body has to feed something the function asserts.

Both directions are controls in audit/scanlib_selftest.py — the logging loop credits nothing, and the accumulate-then-assert loop credits exactly the two features its list names — so the rule cannot be tightened into dropping the real case or loosened back into accepting the logging one.