fix(moss-cli): keep newlines in quoted csv fields on doc import - #488
fix(moss-cli): keep newlines in quoted csv fields on doc import#488RyanRana wants to merge 1 commit into
Conversation
Codex reviewNo issues found. |
|
@RyanRana sign the CLA #488 (comment) |
|
done! |
There was a problem hiding this comment.
Pull request overview
Fixes moss doc add -f file.csv CSV importing so that newlines inside quoted CSV fields are preserved (instead of being lost via splitlines()), while improving CSV validation and error reporting in the moss-cli import path.
Changes:
- Parse CSV from a stream (
io.StringIO) instead ofsplitlines()so quoted multi-line fields remain intact. - Validate/normalize the CSV header (trim whitespace, require
idandtext) and improve error messages with real CSV line numbers. - Add pytest coverage for multi-line fields, header trimming/validation, and line-numbered error reporting.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| packages/moss-cli/src/moss_cli/documents.py | Switch CSV parsing to a stream, validate headers, and report CSV line numbers; refactor JSON parsing for metadata/embedding columns. |
| packages/moss-cli/tests/test_documents.py | Add tests covering multi-line quoted fields, header trimming, missing columns, short rows, and line-numbered errors. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
csv.DictReader was fed content.splitlines(), so newlines inside quoted fields were dropped and words on either side were joined. parse from a stream instead, validate the header, strip a utf-8 bom, and report real file line numbers in errors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
59c012a to
27cb157
Compare
|
Pushed a fix and squashed to a single commit. The blocking codex-review finding was correct — the All required checks are green. Locally, @r4ghu @ashvathsureshkumar review requests are still pending — ready for another look. @HarshaNalluru your review was dismissed by the force-push; I do not have permission to re-request it from a fork, so flagging here. |
moss doc add -f file.csvfedcontent.splitlines()tocsv.DictReader, so a newline inside a quoted text field was dropped and the words around it got glued together ("line oneline two"). now parses from a stream, validates the header, strips a utf-8 bom, and reports real line numbers. tests added.