Skip to content

feature/persist technique tags#1719

Open
patriciapampanelli wants to merge 3 commits intoNVIDIA:feature/technique_intentfrom
patriciapampanelli:feature/persist-technique-tags
Open

feature/persist technique tags#1719
patriciapampanelli wants to merge 3 commits intoNVIDIA:feature/technique_intentfrom
patriciapampanelli:feature/persist-technique-tags

Conversation

@patriciapampanelli
Copy link
Copy Markdown
Collaborator

Summary

  • Persist ordered demon:* probe technique tags on every attempt row as technique_tags.
  • Emit eval_technique aggregate rows mirroring the existing eval_intent report pattern.
  • Preserve technique_tags: [] as a first-class state for probes without technique tags.
  • Preserve T&I metadata when buffs derive new attempts.

Test plan

  • python3 -m pytest tests/test_attempt.py::test_json_serialize tests/test_attempt.py::test_attempt_technique_tags_serialize_and_copy tests/test_internal_structures.py -k "technique_tags or eval_technique" -v
  • python3 -m pytest tests/test_attempt.py tests/langservice/probes/test_probes_base.py tests/test_internal_structures.py -v

Notes

Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Copy link
Copy Markdown
Collaborator

@jmartin-tech jmartin-tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per separate discussion, since tags are part of probes and the probe is already referenced by the attempt, this can be refactored to read these values from the plugin cache and should likely inject a copy of the plugin metadata into the report.jsonl for use in reprocessing digests.

This avoids duplications in the attempts and also provides for easier maintenance and detection of comparability for runs.

Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
@patriciapampanelli
Copy link
Copy Markdown
Collaborator Author

Per separate discussion, since tags are part of probes and the probe is already referenced by the attempt, this can be refactored to read these values from the plugin cache and should likely inject a copy of the plugin metadata into the report.jsonl for use in reprocessing digests.

This avoids duplications in the attempts and also provides for easier maintenance and detection of comparability for runs.

Updated this to stop persisting technique_tags on each Attempt. The eval_technique aggregate now resolves demon:* probe tags via PluginCache.plugin_info(probe_classpath) at evaluator time, using the existing probe identity carried by the attempt.

This keeps plugin metadata in the cache as the source of truth. When #1720 adds the persisted plugin cache header to report.jsonl, this lookup remains the single place to adapt without reintroducing per-attempt metadata.

@patriciapampanelli patriciapampanelli changed the title Feature/persist technique tags feature/persist technique tags Apr 30, 2026
@patriciapampanelli patriciapampanelli moved this from In Progress to In Review in garak / Context Aware Scanning Apr 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

2 participants