Detect encoding damage in labels and synonyms

[gaurav]

Context

Babel has no encoding validation anywhere — no unicodedata, no mojibake detection, no
normalization. Meanwhile three ingest points read their source with a single-byte codec:

• src/datahandlers/datacollect.py — PubChem CID-Title.gz / CID-Synonym-filtered.gz as latin-1
• src/datahandlers/unii.py — UNII_RECORDS_ENCODING = "windows-1252", plus a second file as latin-1

Reading UTF-8 bytes that way produces mojibake (é → é), and nothing downstream notices, because
mojibake is valid UTF-8 — just wrong text. It flows through the compendia and the synonyms files
and lands in Node Normalization as if it were a real name. The one place Python's codec is touched
on the way out (duckdb_exporters.py) raises on invalid UTF-8, which this isn't.

What this adds

Layer	Where	Behavior
Detector	src/synonyms/encoding.py	round-trip + suspect-character test
Pipeline check	5 load sites in src/node.py	raises
Survey tool	babel-check-encoding	reports, exits 1
Backstop	check_for_encoding_issues in duckdb_reports.py	writes reports/duckdb/encoding_issues.tsv

The detector

Non-ASCII text is re-encoded to the codec that would have damaged it and decoded as UTF-8. If that
round-trip succeeds and changes the string, it was mojibake — and the repair is reported, which is
what makes the error actionable (é alone is a puzzle; "probably meant to be 'é'" is a
diagnosis). Legitimate non-ASCII fails one of the two steps, so Ménière disease and
Nα-acetyl-L-lysine stay clean. Secondary signals: C0/C1 controls, U+FFFD, a stray BOM.

Both cp1252 and latin-1 are tried. They damage bytes 0x80-0x9f differently, and only the
matching one can name the original:

PUBCHEM.COMPOUND:3  'N,Nâ\x80\x93dimethyl'
    looks like mojibake (UTF-8 read as latin-1); probably meant to be 'N,N–dimethyl'

An isascii() gate keeps this affordable at full-build scale — nearly every biomedical label is
pure ASCII and exits in nanoseconds, so only the rare non-ASCII string pays for the round-trip.

Why the load side, and why it raises

There is no single write choke point: twenty-odd datahandlers each write their own
babel_downloads/<PREFIX>/labels with a bare write(f"{curie}\t{label}\n"). There is a single
read choke point — the same one SynonymFilter already hooks into. Checking there means each string
is validated once per prefix load, and the error names the file the damage lives in.

Raising rather than warning was a deliberate call: a warning scrolls past in a build log, and the
bad name ships. Re-running one download rule is far cheaper than finding it in a release.
encoding_check_enabled and encoding_check_allowlist in config.yaml are the escape hatches, so
a false positive doesn't need a code change to work around.

No separate write-side pass in write_compendium() — everything it writes already arrived through
one of the five load sites, so it would re-check the same strings and report nothing new.

⚠️ Before merging: the survey has not been run

The check is armed but unvalidated against real data. There's no local babel_downloads, and
pulling a snapshot is a large download I didn't want to start unprompted. A full build could halt on
PubChem or UNII. Please run one of these first:

uv run babel-check-encoding --recursive babel_downloads --out-tsv encoding_issues.tsv

against a stars.renci.org snapshot — or set encoding_check_enabled: false for the first build
and read reports/duckdb/encoding_issues.tsv instead.

If that shows real damage, fixing the latin-1 reads is a follow-up PR, not this one: changing
PubChem's decode alters millions of records, and AGENTS.md is clear that a parsing change needs
the whole file characterized first. This PR produces exactly that evidence.

Also out of scope: NFC/NFKC normalization. That's a deduplication concern, not a corruption one.

Testing

385 unit tests pass; ruff, snakefmt, and rumdl are clean; check_for_encoding_issues resolves
in a snakemake --dry-run. New coverage in tests/synonyms/test_encoding.py,
tests/tools/check_encoding/, tests/node/test_synonym_factory.py, tests/reports/test_duckdb_reports.py
— weighted toward the false-positive cases, since those are what would halt a build.

One bug worth surfacing: I first treated an embedded tab as damage. NodeFactory.load_extra_labels()
splits with maxsplit=1 specifically to preserve tab-containing labels, and
test_load_extra_labels_tab_in_label pins that. The full suite caught it; tab is now explicitly
excluded, with a comment pointing at the loader.

🤖 Generated with Claude Code