Skip to content

test(bench): add cospec-vs-openspec benchmark suite - #31

Open
replygirl wants to merge 15 commits into
mainfrom
worktree-bench-cospec-vs-openspec
Open

test(bench): add cospec-vs-openspec benchmark suite#31
replygirl wants to merge 15 commits into
mainfrom
worktree-bench-cospec-vs-openspec

Conversation

@replygirl

@replygirl replygirl commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • New packages/bench harness for benchmarking cospec against bare OpenSpec: Agent SDK headless runs, hermetic sandboxes, 16 scenarios (including 5 hard variants), hidden held-out tests as the primary escaped-defect metric, arm-blind adversarial review, planted-bug catch rate, DeepSeek quality judge, repeats + paired sign-test stats, resume/publish/backfill modes, and 339 unit tests.
  • Restructures the mise monorepo tasks into per-project mise.toml config roots with back-compat root aliases.
  • Commits RESULTS.md + a README block from the full 192-cell x 2-arm x 2-model run, with a provenance banner.

Headline findings (see RESULTS.md for full detail)

  • Both arms land at ~zero escaped defects on the hidden held-out tests.
  • 0/60 planted-bug catch rate in both arms.
  • cospec runs at roughly a 25% cost premium at equal measured correctness.
  • DeepSeek quality judge scores both arms near ceiling.
  • Net: honest ROI framing — cospec's structure didn't show a measurable quality edge in this run, at extra cost.

Notes

  • Both cospec changes backing this work were archived in-branch, per repo flow (archive is the final commit before merge).
  • Results were published from this branch (provenance banner in RESULTS.md) and can be re-stamped from main post-merge via --publish-from.

Test plan

  • mise run //packages/bench:test — 339 pass
  • mise run typecheck — green across all projects
  • hk pre-push hooks (oxlint, oxfmt, taplo) — passed on push

🤖 Generated with Claude Code

replygirl and others added 15 commits July 21, 2026 19:58
@replygirl @claude
test(bench): add cospec-vs-openspec benchmark harness
fcd7eae 
Add an advisory benchmark that drives a real headless Claude Code agent
(via @anthropic-ai/claude-agent-sdk) through an identical task prompt on
both the cospec arm and the bare-openspec arm, across all 11 schema types
and two models, then scores mechanical validation, artifact
proportionality (from canon type-facts, never hardcoded), and an optional
DeepSeek-judged quality rubric.

New private workspace packages/bench: matrix/CLI parsing, per-cell
hermetic mkdtemp sandbox (cospec arm inits via the working-tree CLI,
openspec arm via the pinned @fission-ai/openspec binary resolved by path),
agent driver, mechanical scorer, DeepSeek judge, and redacted JSONL +
aggregate + markdown reports with a sentinel self-check before every write.

Both arms enable skills: 'all': each tool ships its entire agent-facing
guidance as .claude/skills/<tool>-* (neither writes a CLAUDE.md/AGENTS.md),
so under the hermetic settingSources: [] a symmetric skills setting is the
only way to give each arm exactly its own tool's guidance — enabling it for
one arm only risked running the openspec arm with no guidance at all.

Wire packages/* into workspaces/mise; add advisory bench/bench:smoke tasks
and a check-gated test:bench unit suite (85 tests); add the bench commit
scope. Document in .agents/shared.md (synced to CLAUDE.md/AGENTS.md),
CONTRIBUTING.md, and docs/bench.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): frame both arms toward the spec-driven workflow
c8d821f 
Neither arm's scenario prompt mentioned a spec-driven workflow, so the
agent had no reason not to solve the task directly, measuring nothing
about either tool. Append a byte-identical, tool-neutral instruction to
both arms' SDK systemPrompt (preset claude_code + append, which keeps
Claude Code's default prompt) pointing the agent at .claude/skills.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
build: split mise tasks into per-project config roots
803b47a 
Move project-scoped tasks (test/build/docs/bench/eval) into each config
root's own mise.toml, adding apps/docs, packages/bench, and e2e (a new
config root) alongside apps/cli. Root keeps thin forwarding aliases by
monorepo address so every existing `mise run <name>` invocation, CI
workflow, and hk hook works unchanged. Anchor bun test paths with ./ to
avoid the substring-filter pulling fixture specs into the run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): retune scenario turn budgets for full workflow cycles
2e0de58 
The ci smoke run showed the agent completing propose->author->validate->
implement in 31 turns without archiving, hitting error_max_turns at 30.
Turn budgets were sized for direct implementation, not the full spec-driven
cycle. Raise lite-type scenarios (build, chore, ci, docs, style, test) 30->70
and full-type scenarios (feat, fix, perf, refactor, revert) 60->120. Also
raise the per-cell maxBudgetUsd default 5->10 since full-cycle opus cells can
now exceed $5.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): score archived changes and snapshot artifacts
262eff0 
Post-hoc `cospec validate` only resolves ACTIVE changes by slug, so every
archived cell in the first full run silently scored `cospecValidate: null`.
Validate archived changes via a disposable scratch repo instead, snapshot
each cell's resolved artifacts (redacted) before sandbox teardown for
offline re-scoring, and fix a redaction self-check false positive triggered
by sentinels whose label embeds their own value.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): surface judge failures instead of silent quality: null
81b0917 
Root cause of quality: null on all 44 cells in the first full run: the
configured DEEPSEEK_API_KEY has no account balance, so every judge call
returned HTTP 402 (confirmed by calling the real DeepSeek endpoint from a
throwaway scratchpad script — no code bug in endpoint/model/response
parsing). The real defect was that judgeArtifacts collapsed every sample
failure (HTTP error, finish_reason: length, parse miss) into a bare null,
so a judge outage was indistinguishable from "nothing to judge." judge.ts
now returns { quality, error? }, and a non-null error propagates onto a
new CellResult.judgeError field so this can't be invisible again.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): add held-out hidden/planted suites and hard-mode variants
c67383b 
Hidden bun:test suites (scenarios/hidden/<id>/) and planted latent bugs
(scenarios/planted/<id>/) live outside every fixtureDir so the sandbox
seeder never copies them — the escaped-defect and verification-discipline
signals the agent cannot see. Adds opt-in multi-file -hard variants of the
5 heavy types with their own hidden suites and plants.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): schema-conformance relabel, repeat stats, and diff review
790e3fe 
Relabels the post-hoc cospec-validate metric from "defects" to schema
conformance — cospec's own rubric applied to both arms, never the headline
defect signal (that is escaped defects, from the hidden suites). Adds
scoreHiddenTests/scorePlantedBug wiring, a pure stats module (repeat
spread + paired sign-test comparisons), and an arm-blind adversarial diff
review stage (K=2 reviewers, refute-then-confirm) with per-cell diff
persistence gated behind --review/--review-report. scoreHiddenTests and
scorePlantedBug remove their injected dirs in a finally so a scenario's own
bare `bun test` can never pick up the foreign suites.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
docs(bench): document new scoring signals, stats, review, and hard mode
a7a0212 
Documents escaped defects (primary defect metric) and planted bugs, the
schema-conformance relabel and its framing as cospec's own rubric applied
post-hoc to both arms, repeat statistics, the arm-blind adversarial review
stage, and the opt-in --hard variants with a cost warning. Records the work
in the bench-cospec-vs-openspec change's proposal and tasks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): add committed results publishing (--publish/--publish-from)
6d70676 
Render two COMMITTED artifacts from a finished run's aggregate:
packages/bench/RESULTS.md (full document) and an idempotent managed block
in the root README.md between bench:start/bench:end markers. --publish runs
after a live matrix; --publish-from <dir> re-renders from a past report's
aggregate.json with no agent re-run. Rendering reuses report.ts's
renderSummaryBody/reduceGroup rather than duplicating. Provenance header
carries the commit (org/repo from the origin remote at runtime), tag,
publish date, Claude Code version, models/arms/cells, judge status, and a
non-blocking WARNING banner on a dirty tree or non-main branch. Every write
passes the existing redaction self-check. README is not pre-populated with an
empty block; the append path is covered by tests instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): support resuming interrupted runs (--resume)
3d9d8de 
Add --resume <reportDir> to matrix.ts/run.ts so an interrupted bench run
can be picked back up instead of re-running from scratch: cells already
present in the report's cells.jsonl are skipped (matched by cellKey,
which is per-repeat), only the missing cells run, new results append to
the same cells.jsonl, and aggregate.json/summary.md are regenerated over
the full old+new set. --review on resume backfills review onto
already-complete cells missing reviewDefects, without re-reviewing ones
that already have it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(bench): backfill judge scores from snapshots (--judge-report)
3374fdd 
Add a standalone `--judge-report <reportDir>` mode that re-scores past
cells whose quality came back null by rebuilding the judge input from
each cell's persisted artifact snapshot, with no agent re-run. Used it
to backfill the 80 cells the first full run lost to DeepSeek's HTTP 402
(no account balance) outage, then republished RESULTS.md/README.md with
complete quality data.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
build(archive): archive mise-monorepo-tasks
60d5c12 
Move completed per-project mise.toml wiring change to the archive; task
4.2 (agents:sync) re-verified as a no-op before archiving.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
test(archive): archive bench-cospec-vs-openspec
f0fa452 
Move the completed real-agent bench (cospec vs. openspec, quality-judged
and republished with backfilled scores) to the archive.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@replygirl @claude
build(archive): fix prose-wrap in archived mise-monorepo-tasks tasks.md
30c646e 
oxfmt caught a line-wrap violation in the already-archived tasks.md
(introduced by the 4.2 evidence edit before archiving); reflow only, no
content change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings July 22, 2026 00:59
Copilot AI reviewed Jul 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@replygirl