Skip to content

Normalize null judge detail lists only for positive verdicts #270

Description

@lipikaramaswamy

Problem

LLM judge responses may emit an explicit JSON null for a detail list whose documented empty state is an empty array. Pydantic defaults apply when a field is omitted, but not when it is present as null, so an otherwise usable positive judgment becomes unavailable.

This is related to the model-output drift handled in #268 and the conservative rewrite-evaluator normalization in #269.

Scope

Apply verdict-aware normalization to these evaluation schemas:

  • DetectionJudgmentSchema.invalid_entities
  • TypeFidelityJudgmentSchema.invalid_replacements
  • RelationalConsistencyJudgmentSchema.relations
  • AttributeFidelityJudgmentSchema.entities

Required behavior

Normalize null to an empty list only when the associated verdict is positive:

  • all_valid=true plus invalid_entities=null becomes invalid_entities=[]
  • all_valid=true plus invalid_replacements=null becomes invalid_replacements=[]
  • all_consistent=true plus relations=null becomes relations=[]
  • all_valid=true plus entities=null becomes entities=[]

Keep negative or inconsistent payloads invalid:

  • a negative verdict plus a null or empty detail list must not be accepted as a complete judgment
  • malformed output must continue to surface as judge unavailable rather than fabricating a verdict

Do not normalize nested RelationCheck.entities or AttributeCheck.attributes_checked; an emitted check requires those details.

Acceptance criteria

  • Add model-level validation implementing the positive-verdict-only rule for all four schemas.
  • Add parameterized tests covering positive/null, positive/empty, negative/null, negative/empty, and valid negative/non-empty cases.
  • Preserve the current postprocessing behavior for malformed judgments.
  • No public API changes.
  • make format-check and make typecheck pass.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions