fix: validate dataset save_as file_type and fix save/load round-trip - #763
fix: validate dataset save_as file_type and fix save/load round-trip#763yashwanth123 wants to merge 2 commits into
Conversation
- Raise ValueError for unsupported file_type in Dataset.save_as()
- Accept {"examples": [...]} format in add_from_json/add_from_yaml
so files exported via save_as() can be re-imported
- Reuse example_from_dataset_entry when loading file examples
Co-authored-by: YASHWANTH SAI T <yashwanth123@users.noreply.github.com>
There was a problem hiding this comment.
🔴 Imported examples from user files lose their unique IDs and timestamps
Examples loaded from a JSON or YAML file that omits an id and timestamp are each assigned an empty id and empty timestamp (example_from_dataset_entry at src/judgeval/datasets/dataset.py:72-73) instead of a freshly generated unique id and the current time, so every imported example ends up with a blank identity.
Impact: Importing a plain list of examples (the documented common case) now produces records that all share a blank id and blank creation time, which can collide, overwrite each other, or be rejected on upload.
How the round-trip refactor regressed the bare-list import path
Before this PR, add_from_json/add_from_yaml built each example with Example(name=name), so the dataclass default factories in src/judgeval/data/example.py:53-54 generated a unique example_id (uuid) and a created_at ISO timestamp automatically. Non-name keys became properties.
After this PR both methods route through _examples_from_file_entries -> example_from_dataset_entry (src/judgeval/datasets/dataset.py:97-108). For a user-authored file such as [{"input": "q", "expected_output": "a"}], entry has no example_id or created_at, so entry.get("example_id", "") or "" and entry.get("created_at", "") or "" both evaluate to "" (src/judgeval/datasets/dataset.py:72-73). These empty strings are then passed to the API via example_to_dataset_entry (src/judgeval/datasets/dataset.py:48-53), which lifts example_id/created_at to the top level. The save/load round-trip case still works only because save_as writes the real ids, but the primary documented use of importing plain object lists loses the previously-generated identifiers.
(Refers to lines 71-74)
Prompt for agents
example_from_dataset_entry() in src/judgeval/datasets/dataset.py is now the single path used to build Example objects when importing files via add_from_json/add_from_yaml. For server payloads, example_id and created_at are always present, but for user-authored files (a plain list of objects like [{"input": "q"}]) they are typically absent. The current code sets example_id and created_at to empty strings when absent, whereas the previous import code relied on Example's default_factory to generate a unique uuid and current timestamp. Consider only passing example_id/created_at to the Example constructor when the entry actually contains them (falling back to the dataclass defaults otherwise), so imported examples continue to receive unique ids and timestamps. Be careful to preserve the existing server-payload behavior where those fields are present.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Fixed in 40498ea; example_from_dataset_entry now only sets example_id and
created_at when present in the file. Plain imports get auto-generated IDs.
32 tests passing.
Only pass example_id and created_at to Example when present in the source entry. Plain JSON/YAML imports without those fields now get auto-generated unique ids instead of blank strings. Addresses Devin review on PR JudgmentLabs#763. Co-authored-by: YASHWANTH SAI T <yashwanth123@users.noreply.github.com>
Summary
Fixes two related dataset file export/import bugs.
Bug 1
Dataset.save_as("csv", ...)silently wrote no file. Now raises a ValueError.Bug 2
Files exported via
save_as()could not be re-imported withadd_from_json().save_as writes
{"examples": [...]}but add_from_json expected a bare list.Testing
pytest src/tests/datasets/test_dataset.py — 30 passed