Skip to content

feat(protected-mock): add subset fixture matching and harden mockd startup - #93

Open
dmorosanu wants to merge 1 commit into
codex/uid-gid-agent-isolationfrom
feat/protected-mock-subset-matching
Open

feat(protected-mock): add subset fixture matching and harden mockd startup#93
dmorosanu wants to merge 1 commit into
codex/uid-gid-agent-isolationfrom
feat/protected-mock-subset-matching

Conversation

@dmorosanu

Copy link
Copy Markdown
Contributor

Stacked on #87. Ports the load-bearing pieces of #90 onto #87's protected_mock so #87 becomes the single vehicle and #90 can be closed as superseded.

What is ported

  • match_mode: subset (server.py): token-subset matching evaluated below exact/normalized, scanned in fixture-file order, first match wins, duplicates legal (an earlier rule shadows a later one), empty or noise-only rule argv rejected at load. Real-agent measurement showed finite matching rejects 37% of actual uip invocations over benign extra flags; the skills troubleshoot corpus (Migrate troubleshoot fixtures to protected mocks skills#2503, 297 scenarios) is regenerated against this schema, replay-validated on 379 recorded argvs.
  • mockd startup diagnosability (runtime.py): readiness deadline 5s -> 30s, child stderr captured to a spawner-owned 0600 temp file (outside the socket dir), startup failures report exit code or waited time plus a stderr tail. A loaded box stalls interpreter startup well past 5s.
  • Tokenizer correctness fix (both PRs had it, flagged as P1 in review): the --output noise flag no longer swallows a following flag (deploy --output --delete-all no longer normalizes to bare deploy), and --output=<empty> is dropped atomically instead of consuming the next unrelated token.
  • Doc wording for the two matching modes in TASK_DEFINITION_GUIDE.md.
  • Tests: 4 subset tests ported from feat(sandbox): driver-independent protected mock service for CLI fixtures #90, tokenizer and startup-failure tests are new (7 new tests total).

What is deliberately NOT ported

Validation

  • Windows: ruff, pyright, custom lint, full pytest at baseline (no new failures); 16/16 protected_mock tests.
  • Linux VM: 16/16 protected_mock tests on the branch; real-container e2e under the docker driver with agent isolation - image rebuilt from this branch, a subset fixture, and a flag-decorated invocation (uip rpa get-errors --output json --limit 5 --folder-key demo): the subset rule answered through the full mockd RPC path and the task passed 1/1 (exact/normalized cannot match that argv, so the pass is attributable to the ported matcher). The same task against the pre-port image fails with mockd exiting at fixture load - the old matcher rejects match_mode: subset, which also demonstrates the diagnosability gap the startup hardening addresses.

@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Claude finished @dmorosanu's task in 1m 36s —— View job


Code Review in Progress

Todo List:

  • Read .github/code_review.md for review guidelines
  • Read CLAUDE.md for project conventions
  • Get full diff from base branch (origin/codex/uid-gid-agent-isolation)
  • Read and analyze docs/TASK_DEFINITION_GUIDE.md changes
  • Read and analyze src/coder_eval/protected_mock/runtime.py changes
  • Read and analyze src/coder_eval/protected_mock/server.py changes
  • Read and analyze tests/test_protected_mock.py changes
  • Perform cross-file consistency checks
  • Analyze "what's missing"
  • Provide design-level scrutiny
  • Format and post final review

@dmorosanu

Copy link
Copy Markdown
Contributor Author

Live validation against the real skills #2503 corpus (codex/remove-runtime-seal @ efd1fe6e2) on the Linux VM, image rebuilt from this branch, docker driver with agent isolation, Bedrock claude-sonnet-4-6:

  • 3/3 troubleshoot scenarios SUCCESS (job-stopped-exit-code 1.0, o365-forwardmail 0.925, sap-connection 1.0). Their fixtures are 100% subset rules.
  • mockd's own call log shows 14 fixture answers across 51 uip invocations, of which 10 carried live-only tokens (--output-filter JMESPath projections, --process-name) that no exact/normalized rule could have matched - under the pre-port matcher these return the default empty response and the diagnosis collapses.
  • Probes from the agent uid at +75s: the grader tree, /proc/1/environ, /opt/coder-eval/mock and its fixtures, and the mockd stderr sink are all denied; the legacy in-workspace store (m/.store etc.) is absent on this corpus; no fixture-only strings appear anywhere in the agent workspace.

Two rollout blockers found in the corpus x harness combination (not defects of this diff, filed for the #2503 migration):

  1. The protected_mocks driver validator fires at task LOAD, before experiment/CLI layers merge - all 297 corpus tasks declare no driver and are demoted to skipped; --driver docker cannot fix it. Either the corpus sets driver: docker or the validator moves to a post-merge check.
  2. passthrough_argv_prefixes does shutil.which(tool) at mockd startup and hard-fails; coder-eval-agent images ship no uip, so 238 corpus tasks fail even when the agent never uses passthrough. Needs a soft-degrade or images that include the tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant