feat(artifacts): verify, preserve, and attribute evaluation artifacts - #288
Open
mchillakuru wants to merge 3 commits into
Open
feat(artifacts): verify, preserve, and attribute evaluation artifacts#288mchillakuru wants to merge 3 commits into
mchillakuru wants to merge 3 commits into
Conversation
added 3 commits
July 30, 2026 09:54
finalize_artifact_plan records a sha256 for every output in file_hashes, but the cache activated a version on an existence check alone. An artifact altered after it was written - by an interrupted run, a stray editor save, or anyone with write access to the suite directory - was reused as though it were the computed result, and every score derived from it looked legitimate. _metadata_outputs_exist becomes _metadata_outputs_valid and now compares each output against its recorded hash. A mismatch is a cache miss, not an error: the stage recomputes, which is what the operator wanted anyway, and the mismatch is logged with the file name and both hash prefixes. Metadata without file_hashes predates hash recording and still passes on the existence check, so caches built by earlier versions are not discarded. ASSERT_SKIP_CACHE_VERIFY=1 skips hashing for very large artifacts, at the cost of an unverified cache. _is_local_edit already hashed the suite-root working copies, but that protects hand-edits from being overwritten and never checked the cached artifact itself.
… version Resume and --force-stage called unlink() on inference_set.jsonl and scores.jsonl. Those files are the evaluation evidence - every transcript, every verdict - and a config hash mismatch is not always what the operator intended, so an accidental edit cost a full re-run with no record that anything had been destroyed. archive_artifact() renames the file to <name>.<UTC timestamp>.bak beside the original and logs the path. Backups are never pruned automatically, because pruning evidence is the behaviour being fixed; ASSERT_DISCARD_STALE_ARTIFACTS=1 restores the previous delete. A failed rename falls back to removal with a warning rather than failing the stage. Only metrics.json carried a schema_version, so an artifact written by a newer ASSERT was consumed silently by an older viewer or analysis script. taxonomy.json and manifest.json now carry one inline, and the JSONL artifacts get a .schema.json sidecar. The sidecar is deliberate: a version header inside the JSONL would be parsed as a data record by any reader that does not know about it, including the previous version of ASSERT, so the stamp would corrupt the first row on revert. A sidecar is ignored harmlessly instead. Artifacts with no stamp predate it and are treated as compatible.
Stages hand work to each other through files, and no stage could tell whether its input came from the previous stage, from a cache hit, or from someone editing the file. Only system_prompt_sha crossed a stage boundary; the taxonomy, test set, inference set and scores carried no provenance at all. The schema sidecar added in the previous commit is extended rather than introducing a second mechanism: it now records the producing stage, the assert-ai version, the run id, the target model where relevant, and a sha256 of the artifact's bytes. verify_artifact_provenance() reports when a file no longer matches what its producer recorded, naming the stage. The version is read from package metadata rather than hardcoded, so a provenance record cannot claim a version the running code is not; the CLI's separate hardcoded 0.1.0 is left alone as a follow-up. core.artifact_cache had its own file_sha256 that read whole files into memory. Both now use one chunked implementation in core.io, so the cache digest and the provenance digest cannot drift apart. Unstamped artifacts predate this and still verify, so existing runs keep working.
mchillakuru
requested review from
AaronAspinwall123,
changliu2,
jakepresent and
tangym
as code owners
July 30, 2026 16:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three commits.
Cache integrity.
finalize_artifact_planrecords a sha256 for every output, but thecache activated a version on an existence check alone. An artifact altered after it was
written — by an interrupted run, a stray editor save, or anyone with write access to the
suite directory — was reused as though it were the computed result, and every score
derived from it looked legitimate. A mismatch is now a cache miss, so the stage
recomputes, which is what the operator wanted anyway.
Preservation. Resume and
--force-stagecalledunlink()oninference_set.jsonland
scores.jsonl. Those files are the evaluation evidence, and a config hash mismatchis not always what the operator intended, so an accidental edit cost a full re-run with
no record that anything had been destroyed. They are now renamed aside with a timestamp.
Provenance. Stages hand work to each other through files, and no stage could tell
whether its input came from the previous stage, a cache hit, or someone editing the
file. A sidecar now records the producing stage, the assert-ai version, the run id, and
a digest of the artifact's bytes.
Closes: SEC-005 (cache reused without integrity verification) · SEC-021 (implicit inter-stage trust) · AP-010 (no schema version) · AP-013 (destructive resume)
Commits
Testing
Full suite green on this branch;
9 files changed, 612 insertions(+), 19 deletions(-). Every change has a regression test, and thesuite was re-run after each commit rather than only at the end.
Notes for reviewer
The schema stamp is a sidecar, not a header line, and that is deliberate. A version
header inside a JSONL file would be parsed as a data record by any reader that does not
know about it — including the previous version of ASSERT — so the stamp would corrupt the
first row on revert. A sidecar is ignored harmlessly instead.
Backups are never pruned automatically, because pruning evidence is the behaviour being
fixed.
ASSERT_DISCARD_STALE_ARTIFACTS=1restores the previous delete.Artifacts with no stamp predate this and are still treated as compatible, so existing
runs keep working.
core.artifact_cachehad its ownfile_sha256that read whole files into memory; bothnow share one chunked implementation so the cache digest and the provenance digest cannot
drift apart.
Risk and rollback
Each commit is a single concern and can be reverted independently. See the notes above
for anything that does not revert cleanly.