feat(local): self-introspecting local-agent sandbox evals

[jakepresent]

Why this matters

Agent evaluation is moving from single model-call scoring toward whole-system evaluation: multi-turn workflows, tool use, traces, production monitoring, regression suites, and runtime guardrails. That shift is right, but most current approaches still test slices of the agent. Benchmarks test generic tasks. Observability tools inspect what happened after the fact. Guardrails validate specific controls. Endpoint evals test a model/API surface.

The gap is the configured agent itself. A real agent is not just a model endpoint; it is the runtime plus instruction files, memory, tools, local context, provider routing, credentials, and side-effect boundaries. That is where a lot of behavior comes from, and it is also where many failures only appear.

ASSERT is already close to the right shape because it can turn behavior requirements into generated cases, run a target, judge the results, and preserve evidence. The sandbox direction extends that from “evaluate this endpoint/callable” to “evaluate this configured agent operating environment safely.”

The hard part is safety. If ASSERT generates tests from the agent's actual rules, those tests may ask the agent to use tools, edit files, send messages, or touch sensitive context. We should not point that at the live local profile. The eval target needs to be a copied, disposable runtime with credentials kept out of the sandbox and model calls routed through a controlled host-side proxy.

Summary

This PR adds a local-agent eval path for ASSERT. The technical unlock is self-introspection: instead of ASSERT trying to hardcode every local agent layout, the configured agent describes itself in an agent.yaml; ASSERT snapshots that declared runtime, starts a copied clone in an isolated backend, routes model calls through a host-side auth proxy, builds an ASSERT spec from copied files, and runs the normal ASSERT pipeline against the clone.

High-level flow:

agent self-introspection
-> agent.yaml
-> snapshot declared roots with secret scrubbing
-> materialize sanitized required configs
-> start copied runtime in a sandbox/backend
-> route model calls through host auth proxy
-> smoke configured runtime
-> build ASSERT spec from copied files
-> assert-ai run
-> artifacts/viewer

Major changes

Endpoint target support

• Add OpenAI-compatible endpoint target support with protocol: openai_chat, endpoint URL, model, and env-based bearer auth.
• Preserve the existing simple endpoint contract.
• Capture optional endpoint events as tool-call/tool-result evidence when a target emits them.

Local-agent self-introspection and snapshotting

• Add AgentRuntimeConfig / agent.yaml support for self-described runtimes:
  • roots and external dependencies
  • instruction/persona/memory files
  • launch command
  • endpoint
  • model routing
  • smoke probe
• Add assert-ai local snapshot create --config <agent.yaml>.
• Preserve runtime dependencies by default; drop secrets/churn.
• Support absolute-path excludes emitted by agents.
• Preserve file symlinks by dereferencing them and tolerate live-tree TOCTOU races while snapshotting.

Sandbox/runtime backends

• Keep the proven OpenClaw RAMPART/Docker Sandbox path for stronger isolation and full tool/action evidence.
• Add a generic docker-run backend for faster local dogfood and generic launch proof:
  • same-path mounts for path-bound runtimes like Python virtualenvs
  • local endpoint guard auth
  • host-side auth proxy for provider calls
  • copied runtime config rewrite/materialization
• Add sandbox start --config <agent.yaml> so a self-described config can drive runtime launch.

Required config materialization

Some runtime config files are both required and secret-bearing. This PR adds materialization/patching so ASSERT can avoid copying raw secrets while still producing a runtime-valid config.

Concrete OpenClaw example:

.openclaw/openclaw.json is excluded as secret-bearing
but model_routing.config_file points at that file

The generic materializer can create a sanitized config with endpoint auth, model routing, and provider proxy settings.

Spec build and CLI polish

• Build ASSERT specs from copied sandbox state.
• Preserve sandbox endpoint config in generated specs.
• Infer the single running sandbox for smoke / stop when unambiguous.
• Add command timing output for local-agent commands.
• Keep machine-readable JSON output clean.

What is proven

1. OpenClaw on RAMPART: full-evidence path

This is the strongest demo path today.

discover -> snapshot -> sandbox start -> smoke -> spec build -> assert-ai run -> viewer -> stop

Verified against a copied, configured OpenClaw runtime using live Copilot/gpt-5.5 routing. The RAMPART path captures tool/action evidence from the OpenClaw bridge and remains the full-evidence proof.

2. Hermes on docker-run: second-runtime generic proof

Hermes self-introspected, produced an agent.yaml, snapshotted successfully, booted from the copied/redacted profile through docker-run, routed model calls through the host auth proxy, and completed a Level-1 final-answer assert-ai run.

3. OpenClaw on docker-run: genericity proof against the original bespoke runtime

OpenClaw also completed a Level-1 final-answer run on the same generic docker-run backend after required config materialization. This proves OpenClaw is not only working because of the bespoke RAMPART descriptor.

Validation

Local validation on the current branch:

python -m pytest tests/test_cli.py tests/test_local_sandbox.py tests/test_local_snapshots.py tests/test_local_agent_config.py -q
94 passed, 3 skipped, 2 subtests passed

Full generic Hermes Level-1 run:

POST /v1/chat/completions -> 200, assistant content: HERMES_GENERIC_OK
Target: hermes-agent
Judge: azure/gpt-5.4-mini
Total: 10
Scored: 10
Judge failure rate: 0.0%

OpenClaw docker-run Level-1 run:

/v1/chat/completions smoke: 200, assistant content OPENCLAW_DOCKER_RUN_OK
Target: openclaw
Judge: azure/gpt-5.4-mini
Total: 10
Scored: 10
Policy violation rate: prompt 0.0%, scenario 0.0%
Overrefusal rate: prompt 0.0%, scenario 0.0%
Judge failure rate: 0.0%

Earlier OpenClaw RAMPART proof run produced canonical ASSERT artifacts and viewer output with runtime/tool evidence.

Known boundaries

This is demo/design-review ready, not production complete.

• docker-run is Level-1 final-answer eval. Generic tool/action evidence remains Level 2 and should use the OTel/gen_ai path.
• RAMPART remains the stronger isolation/evidence backend for OpenClaw; docker-run is a faster local dogfood backend, not a RAMPART replacement.
• Non-server/stdio/CLI-only agents are not handled yet.
• Raw artifacts can still contain local paths and need stronger redaction before external sharing.
• Self-introspection should be paired with a future agent-config doctor verifier so ASSERT checks what the agent reported before trusting it.
• This PR is too large to be the final merge shape; if the design direction lands, it should likely be split into smaller reviewable slices.

Review focus

I’m looking for design feedback on:

1. Whether this is the right missing layer between endpoint evals, tracing, guardrails, and production regression.
2. Whether self-introspection is the right product boundary for local-agent evals.
3. Whether docker-run should be treated as an advanced dogfood backend while RAMPART remains the stronger evidence backend.
4. How much of the RAMPART security model should become an ASSERT-owned backend contract.
5. Whether the next priority should be doctor/verification, OTel evidence, non-server agents, or artifact redaction.

[changliu2]

@jakepresent — audit-only observations from a security/readiness pass on 908eaf5 (no approval/changes-requested, just flagging for the local-exec/endpoint work).

Main blocker — endpoint SSRF: local_dev is meant to be the explicit loopback opt-in, but validate_endpoint_url() allows localhost even when local_dev: false, because hostnames in _LOCAL_DEV_HOSTNAMES skip the resolve-and-validate step regardless of allow_localhost. So a normal endpoint target can reach loopback services. Worth fixing before merge, and updating the test that currently asserts localhost is allowed by default.

Two sandbox items before landing local-exec: snapshot staging trusts snapshot_root from the manifest without enforcing it stays under the manifest dir, and stop_local_sandbox() runs cleanup commands read from state JSON — both safer reconstructed/validated from trusted fields than from local JSON.

CI is green and I didn't see shell=True; discovery is read-only and local execution looks opt-in. Might be worth splitting: endpoint target first (after the SSRF + trace-capture fixes), sandbox/local-exec as a separate hardened PR.

(Trace note: endpoint execution doesn't currently wrap target.trace, so the judge won't see OpenInference spans for endpoint targets — flagging for the eval-fidelity side.)

[jakepresent]

Fresh refresh after merging origin/main into this branch (d988fde, includes #236):

• Local targeted suite: python -m pytest tests/test_local_agent_config.py tests/test_local_snapshots.py tests/test_local_sandbox.py tests/test_local_specs.py -q -> 93 passed.
• OpenClaw docker-run e2e against the refreshed branch:
  • local snapshot create: 58,649 files copied, 14,385 excluded.
  • local sandbox start --backend docker-run: started openclaw-main on http://127.0.0.1:18082/v1/chat/completions.
  • direct smoke: HTTP 200 with assistant content OPENCLAW_DOCKER_RUN_OK.
  • local spec build: 5 behavior/context sources.
  • assert-ai run: completed in 5m 1.1s, 10/10 inferences, 10/10 scores, judge failure rate 0.0%, prompt/scenario policy violation 0.0%, prompt/scenario overrefusal 0.0%.
  • local run id: local_agent_openclaw-main_operating_contract/local-agent-refresh-20260615-182504.
• GitHub checks after the push are green: build, CodeQL, Tier 1, Tier 4, and cross-platform install checks all passed.

Still treating this as design/demo evidence, not claiming the PR is ready to merge as one unsliced unit.

[jakepresent]

Fresh RAMPART/Docker Sandbox validation on the same refreshed branch head (d988fde, after merging origin/main):

• Backend: OpenClaw on RAMPART / Docker Sandbox (--backend docker, not docker-run).
• Snapshot from product-shaped commands:
  • local discover --target openclaw
  • local snapshot create --from ... --target openclaw --copy-root ~/.openclaw:.openclaw --copy-root ~/.npm-global/lib/node_modules/openclaw:runtime/openclaw-package --copy-root ~/ChatWorkspace:ChatWorkspace
  • 133,624 files copied, 2,350 excluded.
• Sandbox start:
  • sandbox name: oc-rampart-refresh
  • endpoint: http://127.0.0.1:18081
  • live provider route through host auth proxy to Copilot/gpt-5.5
  • Docker Sandbox launched and then cleaned up.
• Smoke/spec/run:
  • local sandbox smoke completed through the RAMPART endpoint bridge.
  • local spec build: 5 behavior/context sources.
  • assert-ai run: completed in 13m 3.9s.
  • run id: local_agent_openclaw_operating_contract/local-agent-rampart-refresh-20260616-075930.
  • 10/10 inferences, 10/10 scores.
  • judge failure rate: prompt 0.0%, scenario 0.0%.
  • policy violation rate: prompt 25.0%, scenario 0.0%.
  • overrefusal rate: prompt 25.0%, scenario 0.0%.
• Evidence sanity:
  • inference_set.jsonl: 10 rows, ~12.7 MB.
  • raw event/tool evidence strings present (RAW_EVENT_STRING_COUNT=7110).
  • local-path strings are still present in raw inference artifacts (LOCAL_PATH_STRING_PRESENT=True), so the existing artifact-redaction caveat still applies.
• Cleanup verified:
  • docker.exe sandbox ls -> no sandboxes found.
  • endpoint port 18081 free after stop.
  • only the pre-existing OpenClaw POC Docker containers remain running.

This refreshes the stronger RAMPART/full-evidence path separately from the earlier docker-run Level-1 refresh.