Add MP, kept disjoint from HP, with an xref allowlist and HP/MP taxa

[gaurav]

Adds MP (Mammalian Phenotype Ontology) as a disease/phenotype source, kept disjoint from HP
(Human Phenotype): no clique may contain both an HP and an MP identifier. This is the SMEs'
decision after reviewing the overlap-allowed MP addition in #790 — human and mouse phenotypes
behave differently enough that conflating them is undesirable. MP may still merge with non-HP
disease ids (MONDO/MESH); only HP triggers a separation.

MP's xrefs are also restricted to an allowlist of trusted target prefixes (HP, MGI,
MPATH, UMLS). MP uses oboInOwl:hasDbXref to mean "this phenotype is about that thing", not
"is equivalent to it", so nine of the thirteen namespaces it xrefs into are category errors when
read as equivalences. See "MP xref allowlist" below.

This PR also starts assigning taxa to HP as well as MP — every HP identifier becomes
NCBITaxon:9606 and every MP identifier NCBITaxon:40674. HP had no taxa file on main, so this
changes existing output beyond MP; see "Taxa" below.

This is a clean reimplementation off main (curated cherry-pick of #790's substantive MP commits,
dropping churn) plus the new disjointness logic — so #790 (add-mpo) stays untouched as the
"allow overlap" fallback in case this decision is revisited.

Closes #897 — a bug in pull_uber_synonyms() surfaced while re-verifying the MP build after
deleting babel_downloads/MP (see "Review fixes" below).

Closes #906 — a bad MP xref that cliqued "external male genitalia hypoplasia" with "Bifid scrotum"
(see "MP xref allowlist" below). The issue carries executable BabelTest assertions that keep running
against live NodeNorm after it closes.

How disjointness is enforced

glom()'s unique_prefixes cannot do this (it only forbids duplicate same-prefix ids), and
dropping MP's own concord is insufficient (the HP and EFO concords emit direct HP→MP/EFO→MP
xrefs, plus transitive MESH/SNOMED/MONDO bridges). Instead, a post-glom split
(split_mutually_exclusive_cliques, driven by MUTUALLY_EXCLUSIVE_PREFIX_GROUPS = [[HP, MP]])
peels MP out of any clique that also contains HP, keeping the HP side intact. It runs as the last
step of compute_cliques_for_impact_report, the single path shared by the real build and the
source-impact report, so both agree. Precedent: the type-driven split in chemicals.py.
Full writeup + impact: docs/sources/MP/disjointness.md.

MP xref allowlist

MP's UberGraph xrefs are not equivalences. Of the 663 xref rows MP contributes, only 78 survive
review:

Prefix	Rows	The targets are
Fyler	257	Codes from a congenital-heart-defect registry
CL	112	The cell type the abnormality occurs in
MA	85	The mouse anatomical structure that is abnormal
GO	76	The biological process the phenotype perturbs
MGI	70	Kept. MGI phenotype-slim terms
FMA	30	Human anatomy
https, http	18	Wikipedia, Medscape and other web pages
MPATH	4	Kept. Mouse pathology lesions
NLX	4	NeuroLex cell types
UMLS	2	Kept. Phenotype concepts
PMID	2	Literature citations
HP	2	Kept. Genuine phenotype equivalences
NBO	1	A behavior

For example MP:0009873 "abnormal aorta tunica media morphology" xrefs MA:0002903 (the structure
that is abnormal) and MP:0002998 "abnormal bone remodeling" xrefs GO:0046849 "bone remodeling"
(the process it perturbs).

Compendium output is unchanged by the filtering. Every dropped prefix is either absent from all
other disease concords — so NodeFactory drops it at write_compendium, since none of them is in
biolink:PhenotypicFeature's id_prefixes — or, in Fyler's case (221 of its 257 rows), reaches
only HP terms that the [HP, MP] split separates regardless. What changes is that the concord stops
carrying anatomy/process/citation noise, and that a namespace MP newly starts emitting is now
rejected until reviewed rather than silently trusted.

This is implemented as a new allowed_prefixes parameter on build_sets() — the fail-closed
complement of the existing ignore_list, which fails open. Both are matched against
Text.get_prefix_or_none(), which upper-cases, so a non-upper-case entry would silently never
match; build_sets() now raises rather than documenting that landmine. See #907.

One row survives the allowlist and is still wrong: MP:0009203 "external male genitalia
hypoplasia" → UMLS:C0341787 "Bifid scrotum" (a specific malformation that UMLS, HP and SNOMED
agree on). It is dropped explicitly in the new input_data/mp_badxrefs.txt. The [HP, MP] split
already masks it today, so the bad-xref entry is what protects the clique if that policy is ever
relaxed — hence the BabelTest in #906, which asserts both that MP:0009203 stays out and that
HP:0000048/UMLS:C0341787 stay in (a fix that over-splits would be a worse bug). Upstream
reporting to the MP maintainers is tracked in #905.

Taxa — a change to existing HP output

Two new rules (disease_hp_taxa, disease_mp_taxa) derive babel_downloads/<PREFIX>/taxa from
each prefix's already-built ids file, via write_phenotype_taxa and the new disease_phenotype_taxa
config block. Every HP term Babel ingests (the HP:0000118 "Phenotypic abnormality" subtree)
describes a human phenotype and every MP term a mammalian one, so the mapping is a fixed
HP → NCBITaxon:9606, MP → NCBITaxon:40674.

MP's taxa are new output; HP's are a change to output that already exists. main writes no
babel_downloads/HP/taxa, so today every HP-bearing clique has an empty taxa and
taxon_specific: false. After this PR, write_compendium sets each HP identifier's t field and
unions the taxa onto the clique, so:

• every PhenotypicFeature.txt clique containing an HP identifier gains taxa: ["NCBITaxon:9606"]
  and flips taxon_specific to true;
• so does every Disease.txt clique that happens to contain an HP identifier — create_typed_sets
  trusts MONDO's type first, so a MONDO disease clique with an HP member stays a biolink:Disease
  and now claims taxon = human.

That second case is intended, and is worth stating explicitly: HP only ever refers to human
phenotypes, so any MONDO (or other) clique an HP identifier is merged into should be
human-specific. If one isn't, the clique is wrong and we want to find out.

babel-clique-diff compares clique membership only, so none of this appears in the impact
artifacts below — the change is invisible to every number on that page, which is why it gets its
own section here. Downstream exposure is low for now: the taxon fields are really only consumed for
genes and proteins today, and the new NameRes filters by taxon without excluding
taxon_specific: false entries. If the annotation turns out to cause trouble, it can be adjusted
after the next full build.

Impact — how this PR changes Babel

Measured by babel-clique-diff comparing a main build (Babel without this PR — no MP)
against this branch (MP added, disjoint). Both were built the same day from the same cached
babel_downloads, so the UberGraph-derived concords match and the only difference is this PR's
code. (An earlier revision of this page diffed an overlap-allowed build against the disjoint build;
both already contained MP, so the additions were invisible — this baseline avoids that.) See
docs/sources/MP/disjointness.md and disjointness/clique-diff.{csv,summary.json}.

• PhenotypicFeature.txt: 60,718 → 75,469 cliques (+14,751, +24%) — dominated by the 14,750
  wholly new MP-only phenotype cliques. Because they have no main counterpart they appear as the
  summary's clique_count.diff, not as per-clique change rows (the diff iterates
  before-cliques) — which is why the change-row counts below look small next to this number. This
  reconciles with the impact report's ~14,750: the two artifacts just report additions in different
  fields.
• Disease.txt: 365,466 → 365,466 (count unchanged) — MP is a phenotype ontology and adds no
  disease cliques.
• Disjointness holds: 0 cliques contain both HP and MP. In fact MP never coexists with any
  non-MP identifier — all 14,750 MP cliques are MP-only singletons, because MP's surviving xrefs
  (MGI, MPATH) point outside the disease id space and its HP/UMLS xrefs are separated by the
  split. So MP never becomes the preferred leader of a mixed clique.
• Disjointness change rows: 26 MP members regrouped out of HP/EFO phenotype cliques that
  main had leaked them into (via unfiltered HP→MP / EFO→MP xrefs this PR removes); 8 Disease.txt
  cliques released an MP member that moved to PhenotypicFeature.txt; 1 dropped — the stray
  untypeable MP:0005555 (create_typed_sets now drops an untypeable clique with a warning instead
  of aborting the build).
• Incidental reshuffle (not MP membership): 7 Disease.txt cliques (96 members, no MP)
  regrouped. These are cross-references contested between two near-synonymous MONDO cliques that
  unique_prefixes keeps separate (e.g. angiomyolipoma vs kidney angiomyolipoma); MP's presence
  in the build — not its position (verified: MP-last gives a byte-identical diff), and not MP
  membership — shifts which one claims the shared member. Deterministic; creates/deletes no cliques.
  Pre-existing glom behavior, tracked in glom: unique_prefixes tie-break for a contested cross-reference ignores evidence strength #894.
• Not captured above: the taxa change. babel-clique-diff diffs membership, not the t/taxa
  fields, so the HP/MP taxon annotation described under "Taxa" is invisible to every number on this
  page. Read that section alongside these.

What's in this PR

• MP ingestion (cherry-picked from Add MP (Mammalian Phenotype Ontology) as a disease/phenotype source #790): MP prefix + write_mp_ids, SPARQL MP prefix, pipeline
  wiring + xref concords, config lists, disease registered in the source-impact PIPELINE_CONFIG
  with classify_disease_clique, MONDO_close hardening, tests/fixtures, MP docs.
• HP/MP taxa (cherry-picked from Add MP (Mammalian Phenotype Ontology) as a disease/phenotype source #790): write_phenotype_taxa, the disease_hp_taxa /
  disease_mp_taxa rules, and the disease_phenotype_taxa config block. Changes existing HP
  output — see "Taxa" above.
• Disjointness (new): split_mutually_exclusive_cliques + constant + the untypeable-clique
  fix, with unit tests.
• MP xref allowlist (new): allowed_prefixes on build_sets(), MP_XREF_ALLOWED_PREFIXES in
  diseasephenotype.py, and input_data/mp_badxrefs.txt wired into DEFAULT_BAD_XREFS and the
  disease_compendia rule. Unit tests for both; docs/sources/MP/mappings.md documents the
  per-prefix reasoning.
• clique-diff enhancement (new): --before-label/--after-label/--note write a
  self-describing about block, and the per-compendium counts nest clique_count
  (before/after/diff/diff_percent). Docstring + docs/tools/README.md now spell out that wholly
  new cliques appear only in clique_count.diff, never as change rows.
• Impact artifacts: regenerated docs/sources/MP/impact-report* and the
  docs/sources/MP/disjointness/ clique-diff CSV+JSON (now the overall main-vs-branch diff).

Follow-up issues (out of scope here)

Pre-existing behaviors surfaced during review; none blocks this PR:

• get_config() ignores Snakemake --config overrides (e.g. output_directory), splitting the config in two #893 — get_config() ignores Snakemake --config overrides, so redirecting
  output_directory to build into an isolated directory (useful for these side-by-side diffs)
  doesn't fully take effect; compendia still write to babel_outputs/.
• glom: unique_prefixes tie-break for a contested cross-reference ignores evidence strength #894 — glom's unique_prefixes tie-break for a contested cross-reference is sensitive to
  the input set (the incidental Disease.txt reshuffle above). Not introduced by this PR.
• Prefix comparisons that can never match: unify Text's three prefix conventions #907 — Text's three incompatible prefix conventions make a prefix comparison that can
  never match easy to write and impossible to notice. Guarded at build_sets()'s boundary here;
  the general fix (and the rest of the audit — norm()'s other_prefixes keys, build_sets's
  concordfiles keys, ignore_list=["ICD"]) is tracked there.

Review fixes

Addressed in follow-up commits after Copilot and manual review passes:

• Copilot: replaced bare print() progress output with the module logger, removed leftover
  hard-coded debug scaffolding, gave split_mutually_exclusive_cliques() a safe (non-mutable)
  default argument, narrowed a test's exception handling to the specific transient
  UberGraph/network error types, and fixed a stale comment claiming HP/MP-mixed cliques carry
  both taxa (they can't — that's exactly what this PR's split prevents).
• Manual: DISEASE_UNIQUE_PREFIXES and OVERUSE_FILTERED_CONCORDS didn't include MP, unlike
  MONDO/HP/EFO, so MP identifiers were missing the same-prefix overmerge guard and
  overused-xref filtering; compute_cliques_for_impact_report()'s MONDO_close
  basename-discovery bypassed excluded_sources filtering (a --source MONDO impact-report
  run would still apply MONDO's close-match data); and split_mutually_exclusive_cliques()
  scanned every clique in the whole disease/phenotype dict instead of just the ones touching
  HP/MP. All fixed with regression tests.
• Second manual pass: split_mutually_exclusive_cliques() peeled every prefix after group[0]
  rather than every prefix after the earliest occupied one, which for a group whose first
  prefix is absent would strand the clique's out-of-group members in a singleton (unreachable
  with [[HP, MP]], but wrong); it also compared CURIE prefixes with a raw split(":") while
  build_sets() compares against the upper-casing Text.get_prefix_or_none(), so a lower-case
  group constant (prefixes.ORPHANET) would have silently failed open. Both fixed and tested.
  create_typed_sets() now logs a total alongside the per-clique warnings, so a mass drop is
  visible; DEFAULT_BAD_XREFS is anchored at the repo root so the impact-report CLI works from
  any directory; EFOgraph upper-cases the excluded-prefix set once per query rather than once
  per result row (and one logging.warning there is now the module logger); disease_hp_ids
  gained the retries: 3 its MP twin already had; and clique-diff's diff_percent is null,
  not 0.0, when the before build had no cliques — now documented and tested.
• Prefix-comparison guard: build_sets() now raises ValueError on an ignore_list /
  allowed_prefixes entry that isn't upper-case, instead of silently never matching it. This is
  the narrow, boundary-only slice of Prefix comparisons that can never match: unify Text's three prefix conventions #907 — it covers the allowed_prefixes parameter this PR
  introduces, and every existing caller already satisfies it. Unifying Text's three prefix
  conventions (get_prefix_or_none upper-cases, get_prefix doesn't, recurie lower-cases into
  prefixmap) touches ~30 call sites across anatomy/protein/process and needs its own
  build-vs-build diff, so it stays in Prefix comparisons that can never match: unify Text's three prefix conventions #907.
• Re-verification: deleting babel_downloads/MP and rebuilding from scratch surfaced
  pull_uber_synonyms() (src/datahandlers/obo.py) writing empty per-prefix synonyms files
  (with a spurious "not found" warning) for every UberGraph-derived prefix, MP included — its
  internal dict was keyed by full CURIE instead of by prefix, so the per-prefix membership check
  never matched. Fixed by grouping by prefix (mirroring the already-correct pull_uber_labels())
  and correcting the output to 3 columns; regression test added. Practical impact was muted since
  SynonymFactory also falls back to the (correctly-built) common synonyms file. Closes pull_uber_synonyms() silently wrote empty per-prefix UberGraph synonym files #897.
  Note this touches every prefix in generate_dirs_for_labels_and_synonyms_prefixes, not just
  MP, and turns the old "prefix not found" warning into a ValueError — again mirroring
  pull_uber_labels(), so a future UberGraph hiccup fails the rule loudly instead of silently
  writing an empty synonyms file.

Test plan

• uv run pytest -m unit -q — green (287 passed), incl. the split + untypeable-drop tests, the
  clique-diff self-describing/clique_count and new-clique-only-in-count-delta tests, and the
  review-fix regression tests above, and the MP xref allowlist / bad-xref tests.
• uv run ruff check / ruff format --check / snakefmt --check . / rumdl check . — clean.
• Local disease build + disjointness scan: 0 HP+MP cliques.
• Rebuilt concords/MP against UberGraph and confirmed the allowlist shape: 78 rows
  (MGI 70, MPATH 4, UMLS 2, HP 2), down from 663. Regenerated impact-report/ shows the
  MP identifier count untouched (14,750 new cliques, 0 existing cliques modified or merged); every
  changed new-cliques.csv row loses a junk member and empties its needs_biolink_registration
  column.
• babel-clique-diff main-vs-branch shows the additions (as the count delta) plus only MP
  split/move/drop and the one incidental MONDO tie-break reshuffle (glom: unique_prefixes tie-break for a contested cross-reference ignores evidence strength #894) — no unrelated churn,
  so it doubles as a completeness check on the reimplementation.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Copilot reviewed 35 out of 37 changed files in this pull request and generated 3 comments.