Problem
LLM judge responses may emit an explicit JSON null for a detail list whose documented empty state is an empty array. Pydantic defaults apply when a field is omitted, but not when it is present as null, so an otherwise usable positive judgment becomes unavailable.
This is related to the model-output drift handled in #268 and the conservative rewrite-evaluator normalization in #269.
Scope
Apply verdict-aware normalization to these evaluation schemas:
- DetectionJudgmentSchema.invalid_entities
- TypeFidelityJudgmentSchema.invalid_replacements
- RelationalConsistencyJudgmentSchema.relations
- AttributeFidelityJudgmentSchema.entities
Required behavior
Normalize null to an empty list only when the associated verdict is positive:
- all_valid=true plus invalid_entities=null becomes invalid_entities=[]
- all_valid=true plus invalid_replacements=null becomes invalid_replacements=[]
- all_consistent=true plus relations=null becomes relations=[]
- all_valid=true plus entities=null becomes entities=[]
Keep negative or inconsistent payloads invalid:
- a negative verdict plus a null or empty detail list must not be accepted as a complete judgment
- malformed output must continue to surface as judge unavailable rather than fabricating a verdict
Do not normalize nested RelationCheck.entities or AttributeCheck.attributes_checked; an emitted check requires those details.
Acceptance criteria
- Add model-level validation implementing the positive-verdict-only rule for all four schemas.
- Add parameterized tests covering positive/null, positive/empty, negative/null, negative/empty, and valid negative/non-empty cases.
- Preserve the current postprocessing behavior for malformed judgments.
- No public API changes.
- make format-check and make typecheck pass.
Problem
LLM judge responses may emit an explicit JSON null for a detail list whose documented empty state is an empty array. Pydantic defaults apply when a field is omitted, but not when it is present as null, so an otherwise usable positive judgment becomes unavailable.
This is related to the model-output drift handled in #268 and the conservative rewrite-evaluator normalization in #269.
Scope
Apply verdict-aware normalization to these evaluation schemas:
Required behavior
Normalize null to an empty list only when the associated verdict is positive:
Keep negative or inconsistent payloads invalid:
Do not normalize nested RelationCheck.entities or AttributeCheck.attributes_checked; an emitted check requires those details.
Acceptance criteria