fix(rewrite): normalize null evaluator fields

[lipikaramaswamy]

Summary

• normalize explicit null quality answers to the existing conservative unknown fallback
• normalize explicit null privacy fields to the existing highest-confidence leak fallback
• normalize explicit null comparison scores to the existing conservative zero-score fallback
• preserve strict validation for IDs, enum values, score bounds, and answer coverage

Testing

• uv run pytest tests/engine/test_schemas.py tests/engine/test_evaluate.py -q (77 passed)
• make format-check
• make typecheck
• repository pre-commit checks passed

Contribution checks

• No linked issue: narrowly scoped follow-up to null model-output drift handled in fix(detection): normalize null proposed labels #268.
• Public API impact: none.
• PII fixtures: none.
• Secrets: none.

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge.

What we checked:

• Quality nulls remain conservative: No. QualityAnswerSchema changes null to unknown, which the prompt and comparison rules already support as a zero-score answer.
• Privacy nulls fail closed: No. Null answers become yes and null confidence becomes 1.0. Leak checks use those fields, so null output takes the conservative path.
• Null scores become zero: No. The schema changes null to 0.0, matching the missing-comparison fallback. Utility scoring then includes that conservative zero.

Summary

Rewrite evaluators now turn explicit null model fields into conservative fallback values before normal validation. Strict checks for IDs, enums, score bounds, and answer coverage remain in place.

• Quality answers use unknown for null values.
• Privacy answers use a highest-confidence leak fallback for null fields.
• Comparison scores use 0.0 for null values.

Diagram

sequenceDiagram
    participant M as Evaluator model
    participant P as Response parser
    participant S as Rewrite schema
    participant E as Evaluation
    M->>P: Return evaluator JSON
    P->>S: Validate parsed fields
    alt Quality answer is null
        S->>S: Set answer to unknown
    else Privacy field is null
        S->>S: Set yes, 1.0, fallback reason, or empty evidence
    else Comparison score is null
        S->>S: Set score to 0.0
    end
    S-->>P: Return validated result
    P-->>E: Pass normalized answers and scores
    E->>E: Compute utility and privacy results

Loading

_{Reviews (1) · Last reviewed commit: "fix(rewrite): normalize null evaluator f..."}