Stop redoing per-call setup work, and hand the untyped compendium over as JSONL

[gaurav]

Stacked on #986, not main. Fixes here touch chemicals.py, babel_utils.py and write_compendium's callers — all of which #986 rewrites — so branching off main would have guaranteed conflicts. Main's 15 newer commits only touch check-formatting.yml and diseasephenotype.snakefile in code, so nothing collides.

Three of the five follow-ups from the #984–#988 review. All three are repeated or wasteful work in the hot rules, none needs Rust, and all are behaviour-preserving.

Repeated setup work in write_compendium (commit 1)

write_compendium constructs a NodeFactory and an InformationContentFactory on every call, and the chemical build calls it 8 times — once per entry in config.yaml: chemical_outputs. Neither construction was cached, so both did their full setup eight times per build for byte-identical results:

• get_biolink_model_toolkit() built a bmt Toolkit from raw.githubusercontent.com/biolink/biolink-model/<ref>/biolink-model.yaml on every call — a network fetch plus a linkml parse. A build uses exactly one biolink_version, so the cache holds one entry.
• InformationContentFactory.__init__ read all of icRDF.tsv — 212 MB, 3,940,399 lines — calling curies.Converter.compress() per line. That is ~31.5 M compress() calls per chemical build, seven-eighths of them repeating identical work.

The icRDF load moves to a module-level _load_information_content() so it can be cached per path — a test or tool pointed at a different icRDF file still gets its own. The dict is now shared between factories, which is safe because get_ic() only reads it; self.ic was only ever written during __init__.

Neither function had a test. Adds five for the loader plus a network-marked one for the toolkit cache. The read-once test counts real open() calls rather than asserting object identity, since identity alone would also pass if the loader returned a shared default.

The repr(set) handoff (commit 2)

partials/untyped_compendium is the boundary between untyped_chemical_compendia and chemical_compendia — the largest clique state in Babel, ~20 GB at production scale. It held f"{set(s)}\n" read back with ast.literal_eval: Python source used as a wire format between two rules, which is both the slowest deserializer in the standard library and unreadable by anything that is not Python.

Measured over a replicated anatomy clique state (docs/rust-decision/), the repr(set) round trip costs 2.4× what JSONL does — 11.80 s against 5.00 s at 3,069,240 members. Extrapolated to the chemical pipeline's 256,427,006 CURIEs that is roughly 9.5 minutes of chemical_compendia's 19,643 s — more than an in-process Rust boundary could save there even in principle (#988).

Read and write are extracted into read_untyped_compendium() / write_untyped_compendium() so the format lives in one place and is testable without standing up build_compendia's inputs. Reading a pre-JSONL partial raises with rm <path> in the message: Snakemake will not rebuild a partial newer than its inputs, so a tree carried over from an older checkout arrives here in the old format, and failing inside json.loads would name neither the cause nor the remedy.

One thing I got wrong and the test caught: I first claimed sorting members made the file byte-identical across rebuilds. It does not — line order still follows set iteration. The docstring now claims only what is true (sort | diff works) and says why sorting ~100M lines is not worth fixing.

Also here

A commit from #986's lineage covering combine_unichem's one-prefix-per-file guards, which read_concord_file changed the input to and which had no test.

Deliberately not included

Two follow-ups turned out to be much larger than the issue list suggested, and both carry design decisions worth making explicitly rather than burying in this branch:

• write_compendium deriving output paths from get_config() — 27 call sites across 12 modules, and it derives three output trees (compendia/, synonyms/, metadata/) from cdir. The open question is what replaces it: a directory argument, three explicit paths, or paths threaded through as params: the way reports.snakefile:205-213 already does. That choice shapes all 27 sites.
• Sharding generate_pubmed_concords — download_pubmed writes a done-marker, so the ~1,500-file list is not known at DAG time. Sharding needs Snakemake checkpoints, of which this repo has zero, plus a merge step for the accumulating pmid_status. Not something to introduce blind, without being able to run the pipeline.

Verification

425 unit tests pass; ruff check, ruff format, snakefmt clean. Both caches were mutation-tested — removing @cache breaks the read-once test, and deriving the unichem prefix from pairs[0] alone breaks the mixed-prefix test.

🤖 Generated with Claude Code