Skip to content

fix: validate dataset save_as file_type and fix save/load round-trip - #763

Open
yashwanth123 wants to merge 2 commits into
JudgmentLabs:mainfrom
yashwanth123:cursor/dataset-save-load-fix-5d01
Open

fix: validate dataset save_as file_type and fix save/load round-trip#763
yashwanth123 wants to merge 2 commits into
JudgmentLabs:mainfrom
yashwanth123:cursor/dataset-save-load-fix-5d01

Conversation

@yashwanth123

@yashwanth123 yashwanth123 commented Jul 23, 2026

Copy link
Copy Markdown

Summary

Fixes two related dataset file export/import bugs.

Bug 1

Dataset.save_as("csv", ...) silently wrote no file. Now raises a ValueError.

Bug 2

Files exported via save_as() could not be re-imported with add_from_json().
save_as writes {"examples": [...]} but add_from_json expected a bare list.

Testing

pytest src/tests/datasets/test_dataset.py — 30 passed


Open in Devin Review

- Raise ValueError for unsupported file_type in Dataset.save_as()
- Accept {"examples": [...]} format in add_from_json/add_from_yaml
  so files exported via save_as() can be re-imported
- Reuse example_from_dataset_entry when loading file examples

Co-authored-by: YASHWANTH SAI T <yashwanth123@users.noreply.github.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Open in Devin Review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Imported examples from user files lose their unique IDs and timestamps

Examples loaded from a JSON or YAML file that omits an id and timestamp are each assigned an empty id and empty timestamp (example_from_dataset_entry at src/judgeval/datasets/dataset.py:72-73) instead of a freshly generated unique id and the current time, so every imported example ends up with a blank identity.
Impact: Importing a plain list of examples (the documented common case) now produces records that all share a blank id and blank creation time, which can collide, overwrite each other, or be rejected on upload.

How the round-trip refactor regressed the bare-list import path

Before this PR, add_from_json/add_from_yaml built each example with Example(name=name), so the dataclass default factories in src/judgeval/data/example.py:53-54 generated a unique example_id (uuid) and a created_at ISO timestamp automatically. Non-name keys became properties.

After this PR both methods route through _examples_from_file_entries -> example_from_dataset_entry (src/judgeval/datasets/dataset.py:97-108). For a user-authored file such as [{"input": "q", "expected_output": "a"}], entry has no example_id or created_at, so entry.get("example_id", "") or "" and entry.get("created_at", "") or "" both evaluate to "" (src/judgeval/datasets/dataset.py:72-73). These empty strings are then passed to the API via example_to_dataset_entry (src/judgeval/datasets/dataset.py:48-53), which lifts example_id/created_at to the top level. The save/load round-trip case still works only because save_as writes the real ids, but the primary documented use of importing plain object lists loses the previously-generated identifiers.

(Refers to lines 71-74)

Prompt for agents
example_from_dataset_entry() in src/judgeval/datasets/dataset.py is now the single path used to build Example objects when importing files via add_from_json/add_from_yaml. For server payloads, example_id and created_at are always present, but for user-authored files (a plain list of objects like [{"input": "q"}]) they are typically absent. The current code sets example_id and created_at to empty strings when absent, whereas the previous import code relied on Example's default_factory to generate a unique uuid and current timestamp. Consider only passing example_id/created_at to the Example constructor when the entry actually contains them (falling back to the dataclass defaults otherwise), so imported examples continue to receive unique ids and timestamps. Be careful to preserve the existing server-payload behavior where those fields are present.
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 40498ea; example_from_dataset_entry now only sets example_id and
created_at when present in the file. Plain imports get auto-generated IDs.
32 tests passing.

Only pass example_id and created_at to Example when present in the
source entry. Plain JSON/YAML imports without those fields now get
auto-generated unique ids instead of blank strings.

Addresses Devin review on PR JudgmentLabs#763.

Co-authored-by: YASHWANTH SAI T <yashwanth123@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants