Skip to content

feat: add a manual compendium for hand-curated cliques, EUPATH seed term, and source docs - #887

Open
SkyeAv wants to merge 9 commits into
mainfrom
feat/add-shannon-alpha-diversity
Open

feat: add a manual compendium for hand-curated cliques, EUPATH seed term, and source docs#887
SkyeAv wants to merge 9 commits into
mainfrom
feat/add-shannon-alpha-diversity

Conversation

@SkyeAv

@SkyeAv SkyeAv commented Jul 1, 2026

Copy link
Copy Markdown
Member

Adds a manual compendium for hand-curated identifiers, seeds it with EUPATH:0009259 ("Shannon-indexed alpha diversity data"), supports future multi-CURIE manual cliques via equivalents, and documents both the reusable manual mechanism and the current EUPATH example.

Manual Compendium

  • Adds input_data/manual_terms.ndjson as the source of truth for hand-curated entries.
  • Adds src/createcompendia/manual.py and src/snakefiles/manual.snakefile to materialize per-prefix labels/synonyms files and build Manual.txt.
  • Supports optional equivalents so one NDJSON row can form a single multi-CURIE clique while keeping the primary curie first.
  • Rejects overlapping manual cliques by raising on identifiers that appear in more than one row.
  • Passes manual prefixes through extra_prefixes so novel prefixes survive NodeFactory.
  • The manual compendium is part of the normal build: Snakefile and src/snakefiles/util.py always register Manual.txt in the compendium, synonym-export, report, KGX, DuckDB, and sapbert paths.

Documentation

  • Adds docs/sources/MANUAL/README.md for the reusable manual-ingestion path.
  • Adds docs/sources/EUPATH/README.md for the current EUPATH manual entry.
  • Updates docs/sources/README.md so the new docs are discoverable alongside existing source docs like ENSEMBL and UMLS (the source-doc index is linked from AGENTS.md).

Testing

  • Adds targeted unit coverage in tests/createcompendia/test_manual.py for:
    • NDJSON parsing
    • singleton and multi-CURIE clique construction
    • equivalent deduplication
    • duplicate-identifier rejection
    • labels/synonyms materialization
  • Adds aggregator-registration coverage, consolidated into tests/test_snakefile_util.py alongside the existing duckdb_memory_limit_mb tests.
  • Verification:
    • uv run pytest -m unit --no-cov -q -> 412 passed, 105 deselected
    • uv run rumdl check . -> passed
    • uv run snakemake -c 1 -n manual resolves the manual / manual_compendia / manual_labels_synonyms DAG with no extra config

@SkyeAv
SkyeAv requested review from gaurav and hyi July 1, 2026 19:17
@SkyeAv SkyeAv added documentation Improvements or additions to documentation enhancement New feature or request new data source A new data source to include in Babel testing Related to the test suite or testing infrastructure labels Jul 1, 2026
@github-project-automation github-project-automation Bot moved this to Backlog in Babel sprints Jul 1, 2026
@SkyeAv SkyeAv mentioned this pull request Jul 1, 2026
3 tasks
@SkyeAv SkyeAv self-assigned this Jul 1, 2026

@hyi hyi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@SkyeAv I took a quick look at this PR and it looks good to me, so I am approving it. However, please wait for approval from @gaurav before merging.

@gaurav gaurav added this to the Babel v1.19 milestone Jul 10, 2026
@SkyeAv

SkyeAv commented Jul 20, 2026

Copy link
Copy Markdown
Member Author

@gaurav should I want until I can build BABEL before continuing with this? I think I should probably merge current main to this branch and rebase my changes before merging sometime.

SkyeAv added 2 commits July 22, 2026 12:46
…a-diversity

# Conflicts:
#	.gitignore
#	CLAUDE.md
#	docs/sources/README.md
Merge the branch's unstable-gating/aggregator tests (formerly
tests/test_snakefiles_util.py) into main's tests/test_snakefile_util.py
so src/snakefiles/util.py has a single test module.
@SkyeAv

SkyeAv commented Jul 27, 2026

Copy link
Copy Markdown
Member Author

@gaurav regardless of if you merge the PR please let me know what you end up choosing to do with the unstable code so I can both update this PR and also change the development model moving forward

@SkyeAv SkyeAv mentioned this pull request Jul 27, 2026
8 tasks
@SkyeAv SkyeAv added the structural Changes to the structure of Babel in order to apply DRY and to make future development easier. label Jul 28, 2026
…ble flag)

The manual compendium no longer sits behind --config unstable=true. The
manual.snakefile include, the all_outputs done-marker, and the
compendium/synonym aggregators now always register Manual.txt; the
unstable_enabled() helper, the config["unstable"] toggle, the related
docs wording, and the gating tests are removed.
@SkyeAv SkyeAv changed the title feat: add unstable manual compendium for hand-curated cliques, EUPATH seed term, and source docs feat: add a manual compendium for hand-curated cliques, EUPATH seed term, and source docs Jul 29, 2026
@SkyeAv

SkyeAv commented Jul 29, 2026

Copy link
Copy Markdown
Member Author

@gaurav Dropped the unstable flag — the manual compendium is no longer opt-in. Manual.txt now builds as part of the normal pipeline (the unstable_enabled() helper, the config["unstable"] toggle, the --config unstable=true gating, and the related docs/tests are all removed). Title and description updated to match.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request new data source A new data source to include in Babel Priority: High structural Changes to the structure of Babel in order to apply DRY and to make future development easier. testing Related to the test suite or testing infrastructure

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

3 participants