Found by a DevX spike into on-demand visual runs, reproduced independently.
The bug
lib/snap_diff/integrations/minitest.rb:28 increments the assertion count before delegating, and the active? guard lives inside super (dsl.rb:72):
def assert_matches_screenshot(*args, skip_stack_frames: 0, **opts)
self.assertions += 1 # ← unconditional
super(...) # ← returns false immediately when inactive
So with screenshots switched off, a test whose only assertion is a screenshot reports as a passing test with an assertion:
SnapDiff.configure { |c| c.enabled = false }
def test_only_a_disabled_screenshot
assert_matches_screenshot "nope"
end1 runs, 1 assertions, 0 failures, 0 errors, 0 skips
Nothing was captured, nothing compared, nothing asserted. The spike measured the same at scale: 15 tests × (1 real assert + 1 disabled screenshot) reported "30 assertions, 0 failures". RSpec's matcher returns a literal true the same way.
Why this is a 2.0 blocker
It is the silent-success class this release exists to eliminate — the same shape as a missing baseline passing green, and as 0 verified reading like success. A suite that has quietly had screenshots disabled looks identical to one that is checking them, and the assertion count is exactly what a reader consults to tell the difference.
It also actively undermines the honest-reporting work: #261 counts what was verified, and this counts what was not.
The fix, and the free alarm it buys
self.assertions += 1 if SnapDiff.config.active?
Rails unconditionally prepends ActiveSupport::Testing::TestsWithoutAssertions (active_support/test_case.rb:205), which reports Test is missing assertions: `test_x` . With the guard in place, that fires only for tests whose sole assertion was a disabled screenshot — turning Rails itself into a per-test alarm for "this test is no longer checking anything", at zero cost to us.
Checklist
Related: #269 (the summary line is invisible by default). Both are ways a run can say nothing while verifying nothing.
Found by a DevX spike into on-demand visual runs, reproduced independently.
The bug
lib/snap_diff/integrations/minitest.rb:28increments the assertion count before delegating, and theactive?guard lives insidesuper(dsl.rb:72):So with screenshots switched off, a test whose only assertion is a screenshot reports as a passing test with an assertion:
Nothing was captured, nothing compared, nothing asserted. The spike measured the same at scale: 15 tests × (1 real assert + 1 disabled screenshot) reported "30 assertions, 0 failures". RSpec's matcher returns a literal
truethe same way.Why this is a 2.0 blocker
It is the silent-success class this release exists to eliminate — the same shape as a missing baseline passing green, and as
0 verifiedreading like success. A suite that has quietly had screenshots disabled looks identical to one that is checking them, and the assertion count is exactly what a reader consults to tell the difference.It also actively undermines the honest-reporting work: #261 counts what was verified, and this counts what was not.
The fix, and the free alarm it buys
Rails unconditionally prepends
ActiveSupport::Testing::TestsWithoutAssertions(active_support/test_case.rb:205), which reportsTest is missing assertions: `test_x`. With the guard in place, that fires only for tests whose sole assertion was a disabled screenshot — turning Rails itself into a per-test alarm for "this test is no longer checking anything", at zero cost to us.Checklist
active?integrations/rspec.rb) — it returns a literaltruewhen inactiveRelated: #269 (the summary line is invisible by default). Both are ways a run can say nothing while verifying nothing.