Skip to content
This repository was archived by the owner on Nov 2, 2026. It is now read-only.

fix(wordpress): give standalone-only components complete test evidence - #2971

Merged
chubes4 merged 2 commits into
mainfrom
fix/phpunit-empty-discovery
Oct 21, 2026
Merged

chubes4 merged 2 commits into
mainfrom
fix/phpunit-empty-discovery

Conversation

@chubes4

@chubes4 chubes4 commented Oct 21, 2026

Copy link
Copy Markdown
Member

Part of Extra-Chill/homeboy#16202. Pairs with Automattic/wp-codebox#2570.

Problem

For a component whose tests are all standalone PHP smokes (Extra-Chill/extrachill-community: 28 files under tests/, no *Test.php, no phpunit.xml), every review test returned test_runtime_evidence: invalid_evidence, even when the smokes ran. Two separate gaps in this extension caused it:

  1. Discovery. readDiscoveryResult in wp-codebox-phpunit-adapter.mjs rejected files: []. test-inventory.py documents that "empty PHPUnit discovery is valid when the standalone list is non-empty", but the adapter never let an empty result reach it. The CI reason, internal runtime inventory producer failed … WP Codebox canonical PHPUnit discovery failed, came from this. WP Codebox itself also threw on empty discovery; that's fixed in fix(phpunit): report empty discovery as an empty result Automattic/wp-codebox#2570.
  2. Failure attribution. Failed standalone smokes printed PHP_SMOKE_FAIL:<path> but wrote no test.failure records. With discovery fixed, core then reported failed test command did not expose an observed failed test ID.

Change

  • The adapter accepts an empty files array from discovery. Every element must still be an absolute path.
  • test-runner.sh loads the sidecar writer (--sidecar-writer) and the shared test-failures-adapter.
  • Each failed standalone smoke emits one record through the shared homeboy_test_failure_record_json:
    • test_id is the component-relative path, which is exactly the inventory member ID.
    • message is standalone PHP smoke exited <code>.
    • Emission is a no-op when HOMEBOY_TEST_FAILURES_FILE is unset. If a record can't be written, the failing exit status still stands.
  • Not in this PR: a mixed suite in which PHPUnit also fails still has parse-test-failures.sh rewrite the failures file, so the smoke records get dropped there.

Verification

  • End to end on the extrachill-community checkout: homeboy review test --placement local in an isolated HOME, with this extension and the WP Codebox OpenCode runtime must consume Homeboy's canonical workspace permission root #2570 build.
    • Before: invalid_evidence: … canonical PHPUnit discovery failed. With OpenCode runtime must consume Homeboy's canonical workspace permission root #2570 alone, the next gap appeared: failed test command did not expose an observed failed test ID.
    • After: test_runtime_evidence.status: complete, 26 planned tests, and failed_test_ids holding exactly the 4 smokes that genuinely fail (create-ability-rest-routes, forum-read-abilities-rest, public-voice-wp-ability-schema, subscription-inventory-wp-ability-schema).
  • Smoke tests:
    • wp-codebox-phpunit-bounded-output-smoke.mjs has a new empty discovery case that fails on main.
    • test-runner-file-routing-smoke.sh now asserts the standalone failure record (test_id, test_file, message). It fails with main's runner.
    • Also passing: test-inventory-smoke.sh, full-suite-no-phpunit-smoke.sh, test-shard-replay-smoke.sh, standalone-wordpress-core-root-smoke.sh and wp-codebox-phpunit-managed-selection-smoke.mjs, run locally with HOMEBOY_CORE_DIR pointing at Homeboy main.
    • test-inventory-standalone-smoke.sh hits the same local test_path_resolver import error on main, so CI judges that one.

AI disclosure: root cause, fix and tests by Claude Opus 5.5 (Pi coding agent) at Chris Huber's direction; reproduced and verified locally.

A component whose tests are all standalone PHP smokes got invalid
runtime test evidence on every review test:

- the adapter rejected an empty PHPUnit discovery result, although the
  inventory producer treats empty discovery as valid when standalone
  tests exist; and
- failed standalone smokes wrote no failure records, so core could not
  attribute the reported failures to planned tests.

Accept files: [] from discovery, load the sidecar writer, and emit one
failure record per failed smoke keyed by its inventory member ID.
@chubes4
chubes4 merged commit 6e460c0 into main Oct 21, 2026
9 checks passed
@chubes4
chubes4 deleted the fix/phpunit-empty-discovery branch October 21, 2026 04:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant