Add UID/GID isolation for evaluated agents

[dmorosanu]

What changed

• run evaluated Claude, Codex, and Antigravity processes under a dedicated unprivileged Linux UID/GID with cleared capabilities and no_new_privs
• stage agent-visible plugins as manifest-verified public bundles while keeping task, grader, reference, fixture, and output mounts below a root-only parent
• add a protected mock service running as a separate mockd identity behind a group-gated Unix socket
• support exact or finite normalized command maps; normalized matching ignores output formatting and flag order but never uses subset or substring matching
• permit narrowly typed live passthrough such as [docsai, ask], with no shell, bounded execution/output, and in-memory response caching
• scrub harness-only environment paths, bound process counts, reap and verify removal of residual agent processes, and fail closed for unsupported images or privileged dynamic graders
• document the Docker boundary and protected fixture schema

Why

Encryption or runtime sealing does not protect fixtures when the evaluated agent can invoke the decryptor. The boundary must prevent the agent identity from reading grading material and mock fixture bytes while retaining a narrow, observable CLI behavior channel.

Impact and compatibility

Protected Docker isolation is enabled by default. Images must declare org.coder-eval.agent-isolation=uid-gid-v1. docker.working_dir, extra_mounts, and dynamic privileged criteria (run_command, uipath_eval, agent_judge) fail closed until their audiences or grader sandbox are implemented. protected_mocks is opt-in; UiPath/skills#2503 is the dependent migration. Runtime seal is not part of the protected interface.

Native-Linux container validation is still pending because the supplied test VM was unreachable from the implementation environment.

Validation

• repository-wide Ruff passes
• Pyright: 0 errors (one existing Antigravity warning)
• protected-mock/isolation suite: 13 passed and 49 POSIX-only tests skipped on Windows
• broader focused suite: 88 passed and 55 skipped; two unrelated Windows symlink-privilege failures
• earlier broader local run: 4,103 passed; remaining failures were Windows shell/symlink/live-environment cases

[dmorosanu]

Real-container validation: claude and codex arms

Ran this branch on a native-Linux Docker host (Azure VM) against tasks drawn from runs where the deployed harness had read grading material. Both agent backends, same tasks, same probes. Evidence below is from live containers, not code reading.

The launch seams both drop privileges

docker top during live runs, identical across every container:

UID    COMMAND
0      coder-eval        <- harness
2000   claude            <- evaluated agent (claude arm)
2000   codex app-server  <- evaluated agent (codex arm)
2000   codex-code-mode   <- and its child

This matters because the two backends reach the barrier by different routes: claude via cli_path -> coder_eval_claude_agent.sh, codex via launch_args_override -> coder_eval_drop_privilege.sh. Both were unexercised with a live model before this. Direct check: coder_eval_drop_privilege.sh /usr/bin/id -> uid=2000(agent) gid=2000(agent) groups=2000(agent), capability sets all zero, NoNewPrivs: 1, supplementary groups cleared.

Grading material is unreachable

Probed inside each live container as the agent identity (setpriv --reuid=2000 --regid=2000 --clear-groups). Identical result in every container, both arms:

Path	Result
/opt/coder-eval/grader (ls)	Permission denied
grader/input/task.yaml (staged criteria)	Permission denied
grader/input/context.json	Permission denied
grader/task_dir/RESOLUTION.md	Permission denied
grader/task_dir/check_*.py	Permission denied
grader/plugins/plugin-0/tests/tasks (whole skills checkout)	Permission denied
grader/templates	Permission denied
/work/input/... (the old path)	No such file or directory

A find / -xdev as UID 2000 for grading-artifact names returns nothing under /opt/coder-eval. An adversarial probe task also failed on /proc/self/root symlink traversal, the root harness's /proc/<pid>/fd, and /proc/1/environ, and a canary planted in the answer key appeared nowhere in the workspace, transcript, or artifacts.

The two agents behaved very differently, and both results are informative

Claude did go looking, and was stopped. It ran ls /opt/coder-eval/mock/ and find /opt/coder-eval/mock and got Permission denied both times; it could see only the directory names agent-skills grader mock. Every attempt at protected material was attempted-and-denied.

Codex never tried at all. Across 7 tasks and 99 tool calls, zero commands touched any grading-material path. It read /opt/coder-eval/agent-skills/... - the intentional skill projection - and stopped there. The plausible reading is that it had no reason to go hunting: the skills are where it expects them, so nothing prompts a search. That is worth noting because codex is the backend whose earlier runs produced these reads in the first place.

Neither result alone is proof. Claude's denials show the barrier holds under pressure; codex's silence shows the layout removes the motive. The deterministic probes above are what actually establish reachability, independent of what any model chose to do.

What this does not cover

Content staged into the agent's own workspace via mock_path_dirs (m/.store, m/.log, m/.uip.bin, mocks/calls.log) stays readable - it is chowned to the agent by construction, so no UID barrier can hide it. Claude read m/.store on one task and recovered the fixture manifest while still scoring 1.00; codex did not touch it. Closing that is the protected-mock work, not this boundary.

Also: run_command / agent_judge / uipath_eval criteria are rejected under isolation, which currently excludes 591 of 1156 suite tasks (51%). A grader sandbox for run_command is the gating item before this can be turned on broadly.

Fixes pushed while validating

• 02ca02cd - pin npm's global prefix so the Claude CLI lands at the path the isolation wrapper execs. Without it the image does not build.
• 62cfccd0 - exempt the isolation-managed /work/agent workspace from the reserved-path assertion. Isolation assigns that path and the next check rejected it, so every isolated run aborted before starting a container.
• 0502c1b0 - the build-failure fixture needs a concrete agent type now that agent_isolation defaults to true.
• 7f2b135f - stop the evaluated agent inheriting AWS_BEARER_TOKEN_BEDROCK. The UID barrier cannot hide a process's own environment, and no backend needs to inherit it (Claude sets it explicitly from a resolved Bedrock route; codex uses CODEX_API_KEY). This also prevents an inherited token silently steering a direct-route run onto Bedrock.
• 7d4a41a7 - isolate the codex env assertions from ambient scrubbed credentials.

Suite on Linux after these: 4214 passed, 8 skipped. Remaining failures are environmental on that host (no python alias, POSIX-only symlink cases) or live tests needing endpoint config, and reproduce with the change reverted.