[chubes4]

Summary

Job artifact retention never deleted tool-trace.json (or transcript) artifacts, so uploads/.../datamachine-artifacts/jobs/ grew without bound (~510k orphaned job dirs / ~4 GB on events.extrachill.com). This PR makes the default job-artifact policy cover every artifact JobArtifacts writes, legacy files already on disk included. It also makes scheduled cleanup bounded and resumable, so the backlog drains in batches.

Root cause

RetentionCleanup::cleanupJobArtifacts() → FileCleanup::cleanup_old_job_artifacts( 'indexing_artifacts', $days ). walk_old_job_artifacts() only deleted a *.json file when its JSON contained retention_scope: indexing_artifacts. JobArtifacts writes tool traces (artifact_type: tool_trace) and transcripts with no retention_scope, so they never matched and were never deleted, even though retention show reported a 30-day window. The walk also called glob() over every job dir in one request, which can't finish at 500k dirs.

Approach

• Default policy matches by age, not by an opt-in marker (FileCleanup::job_artifact_matcher()):
  • unscoped/legacy artifacts and artifacts with an unregistered scope expire on the default window (datamachine_job_artifacts_max_age_days, 30d)
  • artifacts carrying another registered scope (datamachine_job_artifact_retention_policies) expire on that scope's own max_age_days
  • fast path: a file newer than every window is kept and a file older than every window is deleted, with no JSON read in either case. With default config, nothing gets read.
  • explicit non-default scope calls (count/cleanup_old_job_artifacts( 'x', … )) keep the old stamped-only behaviour.
• Stamp on write: JobArtifacts adds retention_scope (default scope) to tool-trace and transcript payloads, so new files describe their own retention. The engine_data manifest is unchanged, so scoped-artifact summaries are unaffected.
• Bounded, resumable cleanup: new FileCleanup::cleanup_job_artifacts_batch( $scope, $days, $max_dirs, $cursor, $deadline ):
  • selects the smallest $max_dirs numeric job-dir IDs above $cursor in one streaming readdir() pass with a bounded SplMaxHeap (memory scales with the batch, not the 500k total), then processes them in ascending order until the wall-clock deadline
  • returns next_cursor and wraps it to 0 when a sweep completes
  • RetentionCleanup::cleanupJobArtifacts() stores the cursor in the per-site option datamachine_job_artifacts_cleanup_cursor
  • new filters: datamachine_job_artifacts_cleanup_batch_size (default 100000 dirs/run) and datamachine_job_artifacts_cleanup_max_runtime_seconds (default 60). Daily runs drain the backlog over a handful of runs and stay well ahead of growth (~8k dirs/day), and the task can be run manually to drain faster.
• Directory hygiene: emptied job dirs are removed. An empty dir is only removed when it is stale or this pass just emptied it, so an in-flight atomic write is never pulled out from under its writer. Orphaned .tmp-artifact-* files age out with the job. Non-artifact files keep their dir.
• The task description was updated to say it now covers tool traces and transcripts.

Tests

New tests/job-artifact-retention-smoke.php (34 assertions):

• an old unscoped tool trace is deleted and one inside the window is kept
• stamped transcripts are deleted
• unregistered scopes fall back to the default window, and a registered longer scope keeps its own window
• mixed-age dirs lose only expired files
• batch bound respected (3 dirs per batch, cursor advances, untouched dirs beyond the bound), cursor wraps on completion, deadline stops early, non-numeric dirs ignored
• stale empty dirs are removed and fresh empty dirs are kept
• RetentionCleanup::cleanupJobArtifacts() honours the batch filter and persists/resumes/resets the cursor

tests/job-artifact-hashable-smoke.php now also asserts that the tool-trace payload declares the default retention scope.

How I ran them:

cd plugins/data-machine
php tests/job-artifact-retention-smoke.php        # ALL PASS (34 assertions)
php tests/job-artifact-hashable-smoke.php         # passed (49 assertions)
php tests/file-cleanup-count-primitives-smoke.php # PASS
php tests/job-artifact-surfaces-smoke.php tests/job-artifact-content-resolver-smoke.php tests/artifact-manifest-smoke.php tests/retention-reporting-smoke.php tests/job-status-presentation-smoke.php  # all pass
php -l <changed files>                            # no syntax errors
phpcs (homeboy wordpress ruleset) on changed inc/ files  # clean
homeboy review test data-machine --path <worktree>/plugins/data-machine --placement local --wait --settings-json-file /tmp/dm-test-settings.json

Homeboy smoke suite: 376/377 passed. The one failure is tests/email-multi-inbox-forward-smoke.php ("needs WordPress core for the real PHPMailer library"), which fails the same way on a clean origin/main and isn't related to this change. The PHPUnit phase didn't run because of an environment mismatch: wp_codebox_database_service requires database_type=mysql, from the test settings vs. database_type: sqlite in homeboy.json. With --path at the monorepo root, the review discovers no tests on this branch or on a clean origin/main.

Closes #3937

AI disclosure: implemented by an AI agent (Extra Chill Bot) at Chris's direction.

Finalized outside Homeboy Cook (direct fallback authorized by Chris; Cook blocked by homeboy#18081 lab monorepo gate + local provider auth failure).