fix: validate Example.create() property keys - #758
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if key not in _COMMON_KEYS: | ||
| matches = difflib.get_close_matches( | ||
| key, _COMMON_KEYS, n=1, cutoff=0.75 | ||
| ) | ||
| if matches: | ||
| raise ValueError( | ||
| f"Invalid property '{key}'. Did you mean '{matches[0]}'?" | ||
| ) |
There was a problem hiding this comment.
🟡 Valid custom fields resembling common fields are silently rejected
A custom property whose name closely resembles a built-in field name is rejected (difflib.get_close_matches with cutoff=0.75 at src/judgeval/data/example.py:96-103) even when it is intentional, so an example that the documentation says supports "any custom fields" cannot be created.
Impact: Users who legitimately use field names similar to common ones (e.g. plurals like inputs, contexts, or variants like expected_outputs) get an error and cannot build their example, with no way to override the suggestion.
Close-match heuristic has no escape hatch
The class docstring (src/judgeval/data/example.py:20-21) and create() docstring (src/judgeval/data/example.py:83-85) promise that arbitrary custom keys are allowed ("you can add any custom fields", "**kwargs: Any key-value pairs"). However, create() now raises ValueError for any key that scores >= 0.75 similarity against _COMMON_KEYS. For example difflib.SequenceMatcher ratio for "inputs" vs "input" is 0.909 and "contexts" vs "context" is ~0.93, both above the 0.75 cutoff, so these plausible custom field names are rejected with a "Did you mean" error and there is no opt-out. This also flows into production code at src/judgeval/trace/processors/offline_judgment_span_processor.py:61-64, where user-supplied example_fields are expanded into Example.create() during span export; a near-match key there would now raise inside on_end.
Prompt for agents
In Example.create() (src/judgeval/data/example.py), the typo-suggestion check rejects any kwarg whose name is a close match (difflib cutoff 0.75) to a common field name. This contradicts the documented contract that any custom field is allowed, and there is no way to override the suggestion. Plausible intentional custom keys such as plurals (inputs, contexts, expected_outputs) get rejected. Consider making the suggestion non-fatal (e.g. emit a warning instead of raising ValueError for near-matches, while still hard-rejecting the reserved keys), or provide an explicit escape hatch so users can force a custom key. Also note this validation now runs on user-supplied example_fields via offline_judgment_span_processor.py, where raising during span export could disrupt tracing.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Fixes #460