This repository was archived by the owner on Nov 2, 2026. It is now read-only.
Repository navigation
fix(wordpress): give standalone-only components complete test evidence - #2971
Merged
Merged
Conversation
A component whose tests are all standalone PHP smokes got invalid runtime test evidence on every review test: - the adapter rejected an empty PHPUnit discovery result, although the inventory producer treats empty discovery as valid when standalone tests exist; and - failed standalone smokes wrote no failure records, so core could not attribute the reported failures to planned tests. Accept files: [] from discovery, load the sidecar writer, and emit one failure record per failed smoke keyed by its inventory member ID.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of Extra-Chill/homeboy#16202. Pairs with Automattic/wp-codebox#2570.
Problem
For a component whose tests are all standalone PHP smokes (Extra-Chill/extrachill-community: 28 files under
tests/, no*Test.php, nophpunit.xml), everyreview testreturnedtest_runtime_evidence: invalid_evidence, even when the smokes ran. Two separate gaps in this extension caused it:readDiscoveryResultinwp-codebox-phpunit-adapter.mjsrejectedfiles: [].test-inventory.pydocuments that "empty PHPUnit discovery is valid when the standalone list is non-empty", but the adapter never let an empty result reach it. The CI reason,internal runtime inventory producer failed … WP Codebox canonical PHPUnit discovery failed, came from this. WP Codebox itself also threw on empty discovery; that's fixed in fix(phpunit): report empty discovery as an empty result Automattic/wp-codebox#2570.PHP_SMOKE_FAIL:<path>but wrote notest.failurerecords. With discovery fixed, core then reportedfailed test command did not expose an observed failed test ID.Change
filesarray from discovery. Every element must still be an absolute path.test-runner.shloads the sidecar writer (--sidecar-writer) and the sharedtest-failures-adapter.homeboy_test_failure_record_json:test_idis the component-relative path, which is exactly the inventory member ID.messageisstandalone PHP smoke exited <code>.HOMEBOY_TEST_FAILURES_FILEis unset. If a record can't be written, the failing exit status still stands.parse-test-failures.shrewrite the failures file, so the smoke records get dropped there.Verification
homeboy review test --placement localin an isolated HOME, with this extension and the WP Codebox OpenCode runtime must consume Homeboy's canonical workspace permission root #2570 build.invalid_evidence: … canonical PHPUnit discovery failed. With OpenCode runtime must consume Homeboy's canonical workspace permission root #2570 alone, the next gap appeared:failed test command did not expose an observed failed test ID.test_runtime_evidence.status: complete, 26 planned tests, andfailed_test_idsholding exactly the 4 smokes that genuinely fail (create-ability-rest-routes,forum-read-abilities-rest,public-voice-wp-ability-schema,subscription-inventory-wp-ability-schema).wp-codebox-phpunit-bounded-output-smoke.mjshas a newemptydiscovery case that fails on main.test-runner-file-routing-smoke.shnow asserts the standalone failure record (test_id,test_file,message). It fails with main's runner.test-inventory-smoke.sh,full-suite-no-phpunit-smoke.sh,test-shard-replay-smoke.sh,standalone-wordpress-core-root-smoke.shandwp-codebox-phpunit-managed-selection-smoke.mjs, run locally withHOMEBOY_CORE_DIRpointing at Homeboy main.test-inventory-standalone-smoke.shhits the same localtest_path_resolverimport error on main, so CI judges that one.AI disclosure: root cause, fix and tests by Claude Opus 5.5 (Pi coding agent) at Chris Huber's direction; reproduced and verified locally.