feature/persist technique tags#1719
feature/persist technique tags#1719patriciapampanelli wants to merge 3 commits intoNVIDIA:feature/technique_intentfrom
Conversation
Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
jmartin-tech
left a comment
There was a problem hiding this comment.
Per separate discussion, since tags are part of probes and the probe is already referenced by the attempt, this can be refactored to read these values from the plugin cache and should likely inject a copy of the plugin metadata into the report.jsonl for use in reprocessing digests.
This avoids duplications in the attempts and also provides for easier maintenance and detection of comparability for runs.
Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
Updated this to stop persisting This keeps plugin metadata in the cache as the source of truth. When #1720 adds the persisted plugin cache header to |
Summary
demon:*probe technique tags on every attempt row astechnique_tags.eval_techniqueaggregate rows mirroring the existingeval_intentreport pattern.technique_tags: []as a first-class state for probes without technique tags.Test plan
python3 -m pytest tests/test_attempt.py::test_json_serialize tests/test_attempt.py::test_attempt_technique_tags_serialize_and_copy tests/test_internal_structures.py -k "technique_tags or eval_technique" -vpython3 -m pytest tests/test_attempt.py tests/langservice/probes/test_probes_base.py tests/test_internal_structures.py -vNotes
report.jsonl#1703technique_tagsmeans a pre-change report, whiletechnique_tags: []means a new report where the probe had nodemon:*technique tag.