fix(metadata,cli,frontend): ingest jsPsych CSVs with unescaped quotes

[Mandyx22]

Problem

Some jsPsych experiments export the stimulus column as unquoted HTML containing literal " (e.g. <div class = "EncodingBox">), which violates strict RFC-4180 quoting. Surfaced on a real 1258-file OSF working-memory dataset (osf.io/phxq4) where the tool read 0 files — csv-parse threw Invalid Opening Quote and dropped every file. Both the CLI and the browser uploader were affected.

There were two layers to fix:

1. Read: the parser rejected the file outright.
2. Write: even once read, the data file is copied verbatim, so the malformed quotes would land in the Psych-DS data/ payload and the validator (which also strict-parses CSV) rejected it with CSV_FORMATTING_ERROR.

Changes

• parseCSV now sets relax_quotes: true, so a quote inside an unquoted field no longer throws and drops the file; the HTML is kept intact.
• New parseCSVForWrite helper returns the parsed rows plus a verbatimSafe flag (strict-parse probe). The CLI and frontend use it so:
  • a clean CSV still keeps its exact bytes (written verbatim — unchanged behavior), and
  • a file that only parsed thanks to quote relaxation is re-serialised to well-formed CSV, so the written file is valid.

Result (verified end to end)

	Before	After
Library/CLI read	0 files	all files read
Written data file	verbatim (malformed)	re-serialised, quotes escaped
Psych-DS validation	❌ CSV_FORMATTING_ERROR	✅ 0 errors
Frontend (real DataUpload)	every file error	success, output validates (0 errors)

Clean CSVs are unaffected (still verbatim) — only files that were previously 100% unreadable change.

Tests

• Metadata regression: an unquoted field containing " now parses instead of dropping the file.
• Frontend wiring test: clean → verbatim, malformed → re-serialised and strictly re-parseable.
• Updated the DataUpload mock for the new helper.
• Full suites green: metadata 251, CLI 160, frontend 247.

🤖 Generated with Claude Code

[changeset-bot]

🦋 Changeset detected

Latest commit: fd55433

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@jspsych/metadata	Patch
@jspsych/metadata-cli	Patch
frontend	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[jodeleeuw]

The fix handles quote-only fields but not the common quote+comma case.

relax_quotes: true stops INVALID_OPENING_QUOTE, but it does not stop an inner comma from splitting an unquoted field. jsPsych stimulus HTML frequently contains both a literal " and a , (instruction text, lists, key prompts), e.g.:

rt,stimulus,trial_type,trial_index
300,<p>Press "F", "J" to respond</p>,html-keyboard-response,0

The stimulus's inner comma makes the row 5 fields against 4 headers, so csv-parse throws Invalid Record Length: columns length is 4, got 5 even with relaxation. In parseCSVForWrite the strict probe throws, the catch retries leniently, and the lenient parse throws the same record-length error — which propagates uncaught. Since the loop in processDirectory (packages/cli/src/data.ts) isn't wrapped in try/catch, that aborts the whole run, reproducing the "0 files read" symptom the PR targets. The existing regression test only covers a comma-free stimulus, so this gap isn't caught.

I pushed a target spec documenting it (commit 0dc55d0, packages/metadata/tests/csv-input.stress.test.ts) as test.failing so CI stays green; it will flip to a hard failure once comma-bearing stimuli ingest correctly, prompting removal of the marker.

Note "make it parse" isn't free: the only csv-parse knob that lets these through is relax_column_count, which would then silently mis-split the stimulus into the wrong columns and re-serialise valid-but-wrong CSV that passes Psych-DS validation — trading a loud drop for silent corruption. A robust fix probably needs the field quoted at the jsPsych source or a smarter pre-pass. Worth deciding deliberately before claiming the end-to-end fix.

[jodeleeuw]

Maintainer update (discussed with @jodeleeuw): we merged current main into this branch and pushed a follow-up commit that closes the quote+comma gap this PR's test.failing documented — thanks for adding that marker, it became the acceptance test.

What changed:

• parseCSV now escalates leniency only as far as a file needs: strict RFC-4180 first, then relax_quotes, then a new row-repair fallback. Strict-first also addresses the earlier review concern about relax_quotes being applied unconditionally to well-formed files.
• The repair pass re-reads over-split rows with two rules: a closing quote only counts when followed by a delimiter/end-of-line, and (on retry) a comma followed by a space is treated as literal text — machine-written jsPsych delimiters are never followed by a space, while prose/HTML commas (Press "F", "J") almost always are. A row that still can't be matched to the header is skipped with a warning, so one bad row costs that row, not the file.
• The CLI and web wizard now print a warning whenever a file parsed leniently and gets re-serialised, so rewritten bytes are never silent.
• test.failing is flipped to a passing regression test, with new coverage for the half-quoted variant (quoted field, unescaped inner quotes + commas), the quote-only quoted-field edge case, an irreparable-row-is-skipped case, and a CLI end-to-end test asserting the rewritten output strict-parses with the stimulus intact.
• Changeset updated (adds the frontend bump and documents the leniency ladder + the malformed-file multi-parse tradeoff).

Note for #133: since it's based on this branch, it will need this merged in — happy to help with that rebase.

🤖 Generated with Claude Code