Skip to content

workflow: add a code history stage - #894

Open
zevorn wants to merge 8 commits into
sashiko-dev:mainfrom
zevorn:qemu-history-review
Open

zevorn wants to merge 8 commits into
sashiko-dev:mainfrom
zevorn:qemu-history-review

Conversation

@zevorn

@zevorn zevorn commented Nov 13, 2026 •

Copy link
Copy Markdown

Summary

I added a more global review step to the QEMU review: it retrieves the
history and past review discussions of the source code a patch touches,
and uses them to judge whether the patch is sound.

The new history stage runs on every QEMU review, alongside goal,
implementation, and execution-flow. Before the review,
src/workflows/code_history.rs collects the git history of the modified
code (the commits that wrote the changed lines and the commits that later
fixed or reverted them) and, with the server feature, the lore threads
where those commits were reviewed. The stage checks whether the patch
removes a safeguard, repeats a reverted change, or breaks a constraint
recorded in that history. Each history concern must cite the historical
commits it relies on.

The collector picks paths and mail sources from a per-project profile;
only QEMU has one for now. The series also stops QEMU post-verification
from reporting a finding that verification already validated.

Design: designs/DESIGN_CODE_HISTORY_REVIEW.md.

Test results

Six QEMU RISC-V commits that were later fixed or reverted, each reviewed
three times with and without the history stage, in a repository that
does not contain the later fix.

Case Later fix Without history With history
27f9566dcd98 interrupt mask 7f9ab1dc6c 3/3 3/3
c4e6bc63853c gpr size 409c7d8682 3/3 3/3
dcd028517749 vsstatus UXL a9779d24be 1/3 1/3
4fe8ae09062d mhpmcounter 5a9fa8b582 0/3 0/3
63469ad75dcc IMSIC callbacks 0ef6910e32 3/3 3/3
18645f195789 mstatus MPV/GVA 07cea261f0 3/3 (Medium) 3/3 (High)
Per run, all six cases Without history With history
Findings (mean) 21 20
Input tokens (mean) 34.8M 42.7M (+23%)

Detection is the same. With history, the MPV/GVA regression is rated
High in every run instead of Medium, one root cause is reported once
instead of being split, and findings name the historical commits they
rest on.

@sashiko-bot

sashiko-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

Sashiko review — v2

Commit 2/9 — 32f3fc6d workflows: add a per-project code history collector

  • [MEDIUM] src/workflows/code_history.rs: discussions::fetch_thread_for_commit()
    does not set failed = true when client.search_patch_id() or
    commit_patch_id() fails, and only returns None when both failed is true and
    raw_mails is empty. For commits without a Message-ID or lore Link trailer,
    a transient Lore or git error returns an empty CachedThread that
    discussions::thread_for_commit() writes permanently to the on-disk cache.
    Likewise, when client.fetch_thread() fails but client.search_patch_id()
    returns messages, thread_for_commit() permanently caches an incomplete
    thread missing the merged revision's discussion. Return None whenever a Lore
    or patch-ID lookup fails so transient errors are not cached.

  • [MEDIUM] src/workflows/code_history.rs: discussions::parse_mails() drops
    PatchsetMetadata::is_patch_or_cover from parse_email() and sets has_diff
    from patch.is_some(), while discussions::mail_text() checks mail.has_diff
    before treating a message as a reply and checks mail.in_reply_to.is_none()
    for cover letters. Because parse_email() returns Some(Patch) for any
    non-cover email containing an unquoted diff, reviewer replies that include
    inline diff snippets are formatted with posting_notes() and lose their
    review prose. In addition, v2+ cover letters sent In-Reply-To an earlier
    thread are selected by select_mails() without requiring
    in_reply_to.is_none(), but mail_text() skips cover_text() and formats them
    with reply_text() instead. Check meta.is_patch_or_cover when distinguishing
    patch postings from replies, and match the cover-letter condition in
    mail_text() to select_mails().

  • [MEDIUM] src/workflows/code_history.rs: discussions::sanitize() only runs a
    single-pass literal replace for <history_discussions> and
    </history_discussions>. Nested sequences such as
    </history_<history_discussions>discussions> reconstruct a closing
    </history_discussions> tag after replacement, and whitespace or case
    variants as well as other prompt-framing tags pass through unescaped into
    the cached prompt block. Sanitize untrusted mail headers and bodies with
    crate::prior_context::sanitize_untrusted_discussion_text().

  • [LOW] src/workflows/code_history.rs: resolve_parent() and still_present()
    pass the target revision to git rev-parse and git grep without rejecting a
    leading dash or separating options with --end-of-options. If
    target_commit_sha starts with -, git parses it as a flag rather than a
    revision. Reject target strings starting with - or pass --end-of-options.

  • [LOW] src/workflows/code_history.rs: discussions::commit_patch_id() pipes
    the output of run_git() for git show into git patch-id --stable. Because
    run_git() accepts capped output when git show exceeds MAX_GIT_STDOUT (512
    KiB), commits with diffs larger than 512 KiB produce a hash of the
    truncated diff prefix that cannot match on lore.kernel.org. Skip patch-ID
    computation when git show output is capped.

Commit 3/9 — 7fa95fbc worker: collect code history before a patch review

  • [LOW] src/worker/prompts.rs: Worker::run() checks self.stages with raw s ==
    "history" equality, while resolve_analysis_stages_with_options() normalizes
    stage names with normalize_stage_name() and expands the "analysis" alias
    into all analysis stages. Passing --stages with a spelling like
    "stage-history", "History", or "analysis" schedules the history stage with
    state.code_history set to None, so its skip_if predicate skips it. Normalize
    each stage name with normalize_stage_name() and match both "history" and
    "analysis".

Commit 4/9 — f2f01512 workflows: add a history stage to the QEMU patch review

  • [HIGH] src/workflows/qemu_patch_review.rs: analysis_stage() attaches
    skip_if() when code_history is None, while
    resolve_analysis_stages_with_options() still schedules the history stage in
    the BestEffort parallel batch. In src/workflow/engine.rs,
    execute_parallel_batch() counts skipped stages as successes and only
    returns first_error when successes is zero. If code_history is None and all
    running analysis stages fail, execute_parallel_batch() swallows the error
    and the workflow exits early, marking an unreviewed patch as clean. Omit
    the history stage in resolve_analysis_stages_with_options() when
    code_history is None, or do not count skipped stages as successes in
    execute_parallel_batch().

  • [MEDIUM] src/workflows/qemu_patch_review.rs: render_history_blocks()
    interpolates history.rendered and history.discussions into the
    <history_context> and <history_discussions> XML blocks without passing them
    through sanitize_untrusted_discussion_text(). In
    src/workflows/code_history.rs, discussions::sanitize() only strips exact
    lowercase <history_discussions> and </history_discussions> once, so
    case/whitespace variants, </history_context>, or other prompt framing tags
    in commit messages or mailing list replies can break out of the XML block.
    Sanitize both strings with sanitize_untrusted_discussion_text().

  • [MEDIUM] src/workflows/qemu_patch_review.rs: STAGE_VERIFICATION_INSTRUCTION
    tells verification_stage_with_effort() to route history concerns to
    inconclusive_concerns with signal_reason set to insufficient_evidence when
    historical_commits cite a commit not in <history_context> or rely on mail
    alone. However, <history_context> and CodeHistoryContext::commit_shas are
    never passed to verification_stage_with_effort() or annotated onto the
    concerns, and the schema prompt in verification_stage_with_effort() omits
    insufficient_evidence from the allowed signal_reason values. Tag each
    historical_commits entry against commit_shas in analysis_stage() and list
    insufficient_evidence in the verification schema prompt.

  • [MEDIUM] src/workflows/qemu_patch_review.rs:
    STAGE_POST_VERIFICATION_INSTRUCTION tells post-verification to read every
    commit in the candidate's historical_commits array with git_show and keep
    that array on confirmed findings. However, verification_stage_with_effort()
    and apply_verification_stage_output() do not include or copy
    historical_commits onto state.inconclusive_concerns, and the follow-up
    commit "workflows: keep historical commits on QEMU history findings" only
    copies historical_commits onto state.findings rather than
    state.inconclusive_concerns or state.concerns. Propagate historical_commits
    onto state.inconclusive_concerns in apply_verification_stage_output() and
    onto state.concerns in record_verified_findings().

Commit 5/9 — 7efcf219 workflows: keep historical commits on QEMU history findings

  • [MEDIUM] src/workflows/qemu_patch_review.rs: carry_historical_commits()
    resolves historical_commits for state.inconclusive_concerns into the local
    by_id map without writing them back onto the inconclusive items themselves
    (or onto state.verification_findings and state.post_verification_findings).
    When resolve_post_verification_stages_with_options() serializes batches of
    state.inconclusive_concerns into candidate_inconclusive_concerns,
    post-verification candidates from the history stage lack the
    historical_commits array that STAGE_POST_VERIFICATION_INSTRUCTION tells the
    model to inspect with git_show. Also, checking
    finding.get("historical_commits").is_some() skips populating the field when
    the model emits null or an empty array. Write the resolved
    historical_commits onto each item in state.inconclusive_concerns and only
    skip findings that already have a non-empty historical_commits array.

Full review and stage logs on sashiko.sashiko.dev

@zevorn

zevorn commented Nov 13, 2026

Copy link
Copy Markdown
Author

FYI @alistair23

Describe a review step that judges a patch against the history of the
code it changes. A deterministic collector reads the commits that
wrote the modified and deleted lines, the commits that introduced
identifiers the patch removes, and the commits that later fixed or
reverted them, plus the lore threads where those commits were
reviewed, using a per-project profile for which paths count as code
and where reviews happen.

QEMU runs the history stage on every review. Its concerns must name
the historical commits and the historical fact they rely on; the
commits are grounded in the collected history and carried through
verification to the report. Post-verification also stops repeating
findings verification already validated.

Signed-off-by: Chao Liu <chao.liu@processmission.com>
Add code_history, which reads, as of the parent of the target commit,
the git history of the code a patch changes: the commits that last
wrote the modified and deleted lines, the commits that introduced
identifiers the patch removes, and the commits that later fixed or
reverted any of them. The commits are ranked by how directly they
explain the changed code and rendered with their messages and diffs
in the modified files, within fixed size, command count, and time
bounds. Series commits are excluded and only object names reach git
as revisions.

With the server feature, it also fetches the review threads of the
most relevant commits through the existing lore client, by Message-ID
or Link trailer and by stable patch ID, which also finds earlier
versions. It keeps postings, cover letters, and reply prose about each
commit, parses mail off the async runtime, caches a thread only when
every request behind it succeeded, and escapes tag-shaped text in
commits and mail before it reaches a prompt.

Which paths count as code and where reviews happen come from a
per-project HistoryProfile. Only QEMU has one so far.

Signed-off-by: Chao Liu <chao.liu@processmission.com>
Add a history analysis stage that runs on every QEMU review, beside
goal, implementation, and execution-flow. Before the review, the
worker collects the code history of the target commit, and the stage
judges the change against it as a long-term maintainer would: does it
remove a safeguard whose reason still holds, repeat a reverted
approach, leave earlier occurrences of a fixed defect behind, or break
a compatibility or specification constraint recorded in history? The
stage is left out of the run when nothing was collected, and in cheap
mode it runs beside the combined stage.

Every history concern must name a history category, the historical
fact it relies on, and the historical commits it cites. The cited
commits are grounded in the collected history, taking SHAs and
subjects from git and flagging commits the history did not show, so
verification can route those to post-verification, which reads each
cited commit. The grounded commits are carried onto inconclusive
items, findings, and pre-existing findings from their source concerns
only, and the report names them. Concerns about Fixes trailers, commit
messages, comments, tests, or style stay with the other stages.

The QEMU false-positive guide gains the history-specific cases.

Signed-off-by: Chao Liu <chao.liu@processmission.com>
Post-verification batches run after verification and see neither each
other nor the findings verification validated directly, so a batch
could confirm, under a new ID, a defect already reported; on a real
RISC-V commit one vmstate defect was reported three times. List the
validated findings, with their pre-existing status, in the
post-verification prompt and have it dismiss a candidate with the
same root cause and status and report only the mechanism it was asked
to verify. Have verification classify each root cause once, never
into both findings and inconclusive items.

Signed-off-by: Chao Liu <chao.liu@processmission.com>
@zevorn
zevorn force-pushed the qemu-history-review branch from 8b1b2a5 to 72dff34 Compare November 13, 2026 09:39
@zevorn

zevorn commented Nov 13, 2026 •

Copy link
Copy Markdown
Author

Thanks. All findings of the v2 review are valid and fixed

@sashiko-bot

sashiko-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown

Sashiko review — v3

Commit 2/4 — a16426ed workflows: add a per-project code history collector

  • [LOW] src/workflows/code_history.rs: discussions::commit_patch_id() checks
    diff.len() >= MAX_GIT_STDOUT to skip truncated diffs, but run_git() uses
    capped_output_with_limit(), which trims capped stdout back to the last
    newline before MAX_GIT_STDOUT and drops out.stdout_capped. When a commit
    diff exceeds 512 KiB and does not end on that exact byte boundary,
    commit_patch_id() hashes the cut diff and queries lore.kernel.org with a
    bogus patch ID. Check out.stdout_capped instead of diff.len().

Full review and stage logs on sashiko.sashiko.dev

The patch ID path must tell a cut diff from a complete one, and
run_git drops the cap flag.

Signed-off-by: Chao Liu <chao.liu@processmission.com>
A capped capture is cut at the last newline below MAX_GIT_STDOUT, so a
length check misses it and hashes a partial diff into a patch ID that
matches nothing or the wrong thread.

Signed-off-by: Chao Liu <chao.liu@processmission.com>
@zevorn zevorn changed the title QEMU review: add a code history stage workflow: add a code history stage Nov 14, 2026
The negative case alone would stay green if commit_patch_id returned
None for every diff.

Signed-off-by: Chao Liu <chao.liu@processmission.com>
@zevorn

zevorn commented Nov 14, 2026

Copy link
Copy Markdown
Author

The v3 finding on commit 2/4 is fixed. Code history now keeps the git capture's capped flag (14b9e14), commit_patch_id skips a capped diff instead of comparing its trimmed length (e20892a), and the regression test covers both the capped and uncapped paths (5dc02c8).

@sashiko-bot

sashiko-bot Bot commented Nov 14, 2026

Copy link
Copy Markdown

Sashiko review — v4

Commit 2/7 — a16426ed workflows: add a per-project code history collector

  • [MEDIUM] src/workflows/code_history.rs: collect_code_history() passes the
    unfiltered diff to extract_diff_blame_targets() and only filters the
    returned targets with HistoryProfile::is_code_path() afterward. Because
    extract_diff_blame_targets() stops at MAX_BLAME_HUNKS (8), hunks in
    skipped paths such as docs/, tests/, or meson.build consume that budget
    before being dropped, and if they account for the first 8 hunks,
    collect_code_history() returns None even though the patch modifies code.
    removed_only_identifiers() also scans the unfiltered diff, so identifiers
    deleted in skipped files consume MAX_IDENTIFIER_PROBES while identifiers
    removed from code but added in docs/ or tests/ are skipped. Filter the
    diff to code paths before extracting blame targets and removed
    identifiers.

  • [LOW] src/workflows/code_history.rs: collect_code_history() checks
    targets.is_empty() without checking files.is_empty(). Because
    extract_diff_blame_targets() populates files only from diff --git
    headers while building targets from --- a/ headers as well, a unified
    diff without diff --git headers leaves files empty while targets is
    non-empty. still_present(), collect_removed_identifiers(), and
    render_commit() then pass -- with no pathspecs to git grep, git log
    -S, and git show, running across the whole repository instead of the
    modified files. Return None when files.is_empty() or seed files from
    targets.

Full review and stage logs on sashiko.sashiko.dev

Hunks and identifiers from docs, tests, and build files used up the
blame and probe budgets. A diff without file names made git grep,
git log -S, and git show run without pathspecs.

Signed-off-by: Chao Liu <chao.liu@processmission.com>
@zevorn

zevorn commented Nov 15, 2026

Copy link
Copy Markdown
Author

The v4 findings are fixed in c634172. The collector filters the diff by HistoryProfile::is_code_path before extracting blame targets and removed identifiers, so skipped paths no longer consume the blame hunk or identifier probe budgets. The pathspec list is seeded from the blame target pre-image paths when a diff has no git headers, so git grep, git log -S, and git show no longer run without pathspecs. Tests cover git-header filtering, headerless scoping, and leading headerless sections.

@sashiko-bot

sashiko-bot Bot commented Nov 15, 2026

Copy link
Copy Markdown

Sashiko review — v5

✓ No issues found across 8 commits.

Full review log on sashiko.sashiko.dev

@alistair23

Copy link
Copy Markdown
Contributor

This is a cool idea! Reading the test results though, does this actually catch any issues that wouldn't otherwise be detected?

@zevorn

zevorn commented Nov 16, 2026

Copy link
Copy Markdown
Author

This is a cool idea! Reading the test results though, does this actually catch any issues that wouldn't otherwise be detected?

I'm testing more.

@zevorn

zevorn commented Nov 16, 2026 •

Copy link
Copy Markdown
Author

Summary

I ran this branch against a QEMU series that the baseline had already reviewed, with the same command and settings, and compared the findings.

Tested series

Field Value
Subject [PATCH v6 00/35] hw/arm/smmuv3: Support Secure state for SMMUv3
Author Tao Tang <tangtao1634@phytium.com.cn>
Cover letter <20261114154623.3599750-1-tangtao1634@phytium.com.cn>
Mailing list qemu-devel (cc qemu-arm)
Posted 2026-11-14 23:46 +0800 (cover), 2026-11-15 00:13 +0800 (patches)
Size 35 patches plus the cover letter, 18 files, +1597/-462
Base QEMU master e9fa1f66f0 (2026-11-13)
Branch review/smmuv3-secure-v6, applied from the hkml mbox; 35/35 patch IDs match the author's branch
Lore https://lore.kernel.org/qemu-devel/20261114154623.3599750-1-tangtao1634@phytium.com.cn/

Baseline: sashiko 0.9.1. This PR: head c634172e, built as 0.9.3. Both runs used sashiko review --cheap --project qemu master..HEAD and the same deepseek-flash settings.

The history stage attaches historical commits to five findings on this series. One is a new history-only issue (sbsa-ref platform version, Patch 27). One reconstructs the lineage of a register mask and raises its severity from Low to Medium (Patch 1/26). One raises a baseline Medium to High (Patch 29). One adds contract evidence to a baseline Medium (Patch 28, S_INIT forwarding). One is a new High (Patch 28, S_INIT re-lock).

Test results

Metric Baseline PR #894
Critical 0 0
High 1 2
Medium 6 5
Low 32 23
Total findings 39 30
Input tokens 66.0M 104.0M (+58%)
Output tokens 2.65M 4.07M
Case Baseline PR #894
Patch 1 S_CR0 mask Low Medium
Patch 26 S_CR0 mask, write path Low Medium
Patch 27 sbsa-ref platform version — Low
Patch 28 S_INIT forwarding to host SMMU Medium Medium
Patch 28 S_INIT re-lock — High
Patch 29 secure-window gate Medium High
Patch 17 TLBI invalidation scoped by queue High — (not reproduced)
Patch 19 accel-forward guard — Medium
Patch 24 CMDQ_CONS error recovery Medium Low
Patch 34 secure-impl + accel Medium Low
Patch 11 stage-2 PTW attributes Medium Low

Findings from the history stage

Patch Severity Finding Historical commits Kind
1, 26 Medium SMMU_S_CR0_RESERVED reintroduces the pre-fix mask fae4be38b35d, 65152c2942c3 New evidence, severity raised
27 Low sbsa-ref bumps no platform version for the new secure SPIs 62c2b8760b8e New
28 High S_INIT re-locks s->mutex through event-queue DMA 32cfd7f39e08 New
28 Medium S_INIT.INV_ALL not forwarded to the host SMMU 58a789c389cb Evidence added
29 High Secure-window gate accepts unset MemTxAttrs.space 4d6e1c6495f4, a14503827d32 Severity raised

Patch 1/26, the strongest result. SMMU_S_CR0_RESERVED is 0xFFFFFC12, so RES0 bits 5 and 9 are writable. 65152c2942c3 had already changed SMMU_CR0_RESERVED from 0xFFFFFC20 to 0xFFFFFA20 for exactly this reason, and fae4be38b35d introduced the old value. The new Secure constant brings the pre-fix encoding back. The baseline rated it Low; the history run rates it Medium. The same root cause is reported on Patch 1 and Patch 26, so cross-patch deduplication did not merge them.

Patch 27, history-only. create_smmu() wires secure outputs 4..6 to SPIs 17..19, while machine-version-minor stays 4 and docs/system/arm/sbsa.rst gains no entry. 62c2b8760b8e (XHCI) followed the documented convention: bump the version and register the capability.

Patch 29, severity raised. ARMSS_Secure is the zero enumerator, so MEMTXATTRS_UNSPECIFIED passes the Secure-window check. 4d6e1c6495f4 made all-zero mean "not specified"; a14503827d32 provided arm_space_is_secure() for this kind of model.

Patch 28, evidence added. S_INIT.INV_ALL clears only QEMU-side state and never forwards to the host SMMU. 58a789c389cb established that contract for every other guest invalidation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants