Skip to content

Add validated execution examples to 12 skills and fix bugs they exposed - #52

Merged
bowen-bd merged 12 commits into
learningmatter-mit:mainfrom
bowen-bd:docs/skill-examples
Oct 10, 2026
Merged

bowen-bd merged 12 commits into
learningmatter-mit:mainfrom
bowen-bd:docs/skill-examples

Conversation

@bowen-bd

@bowen-bd bowen-bd commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds validated execution examples to the 12 executable skills that had none. Each example compares its outputs with published literature, an official database release or a theory reference, and gives citations. Running the examples exposed bugs in several skills and in the shared MD code; this PR fixes them.

Examples added

Skill Example Reference Result
chem-db-qmof HKUST-1 (DOTSOV01) QMOF figshare release; Chui et al. 1999 Band gap and density identical to the release; a = 26.474 Å vs 26.343 Å exp. (+0.5 %)
chem-db-mof ARC-MOF ddmof_559 Majumdar et al. 2021; ARC-MOF metadata Volume, density and Ni₄(μ₃-OH)₂ node stoichiometry match
chem-nmr-predict ethyl acetate, ethanol Fulmer et al. 2010; Gottlieb et al. 1997 Ethyl acetate MAE 0.045 ppm; ethanol CH₂ error (0.41 ppm) and missing OH documented
chem-nmr-analysis camphor reduction; kinetics series Lopansri et al. 2022; reference-free rate constant 13.1 % borneol vs 12.6 % direct integral (17 % in the paper; gap is in the digitized input); k within 5 %
general-arxiv-search MACE title lookup; MLIP keyword search arXiv abs records All fields match
general-biorxiv-search DOI lookups; date-range search bioRxiv API, Crossref All fields match
mat-dft-mixing-functionals Fe₂O₃, Al₂O₃, FeS₂ MP corrected energies; Wang et al. 2021 Reproduces MP2020 corrections exactly
mat-elemental-energies Li, Fe, Cu, O, Si × 4 checkpoints MP GGA / r2SCAN energies Within the models' published MAE; Li₂O formation energy within 0.2 meV/atom of MP
mat-md-monitors Cu NVT, 300 K and 100 K Equipartition; Dulong–Petit ⟨ΔE_pot⟩ within 3.5 % / 0.7 %; c_v within 1.6 % of 3R
ml-mlip-nvalchemi batched vs sequential static, strained Cu stored benchmark table ΔF = 0, ΔE ≤ 1.9e-6 eV; batched MD re-measured at 1.40× (stored 4.90× does not reproduce), SKILL.md updated
ml-mlip-speed NaCl up to 2000 atoms, 4 models stored GB10 reference ms/atom within 5–18 % and MB/atom within 14 % (uma-s-1p1 now 4.5× faster)

The skills without examples either execute nothing (rules, setup, references, protocols) or need ORCA, which will be tested on another machine.

Fixes

  • MD (src/utils/mlips):
    • run_md treated every input as a restart because Atoms.get_velocities() returns zeros, not None, so a perfect crystal started at rest and stayed at 0 K.
    • ExplosionMonitor read the caller's copy of the atoms, not the integrated atoms, so it could never fire.
    • run_md now returns status: "stopped" and stop_reason when a monitor ends a run.
    • final_structure is now the last MD frame rather than the input.
    • Explicitly supplied momenta, including all zeros, are kept (no relaxation or velocity redraw), so NVE starts from rest work.
  • chem-nmr-analysis: --baseline-correct subtracts only the minimum, which biases noisy or offset spectra toward 50:50. New --baseline-window (median or max of a signal-free window), --ppm-range with --range-protons (a restricted fit counts only the protons inside the range), a reported noise fraction, and a corrected fit plot. Workflows updated to match.
  • general-arxiv-search: multi-word queries were OR-ed (all:a b c), which returned off-topic papers. Terms are now AND-ed, authors are quoted, and categories are combined correctly.
  • general-biorxiv-search:
    • Multi-word categories never matched.
    • DOI lookup returned v1 instead of the latest version.
    • medRxiv URLs pointed at bioRxiv.
  • chem-db-qmof / chem-db-mof:
    • Refcode lookup returned nothing.
    • Band gaps were never saved.
    • ddmof_42 matched ddmof_4291.
    • --elements now requires all listed elements by default; --element-match any covers mixed-metal sets.
    • SKILL.md described downloads, file names and charges the code does not use.
  • mat-elemental-energies:
    • Corrected O and Cl references where the molecular crystal collapsed during relaxation: MACE-MP-medium, MACE-MH-0 (three heads), MACE-MATPES-PBE-0 and MACE-MP-small.
    • Regenerated the TensorNet r2SCAN and PBE v2025.1 files with the weights now loaded under those names (adds the missing TensorNet-MatPES-PBE-v2025.1-PES alias).
    • The documented command now works from the repository root.
  • ml-mlip-speed: the plot-only step read a different file from the one runs write.
  • Citations: SPINUS (Anal. Chem. 2002), Lopansri et al. 2022 (replacing a citation that could not be found), and Burner et al. 2023 (ARC-MOF).

Known follow-ups (not in this PR)

  • Deprecate CHGNet-MPtrj-2023.12.1-2.7M-PES in favour of CHGNet-PES-MatPES-*-1M-2026.9, and deprecate M3GNet (the M3GNet-MP-2021.2.8-PES alias currently loads MatPES-PBE weights; the M3GNet-MatPES-v2025.1 library files are stale).
  • ORCA examples, to be tested on another machine.

Testing

  • venv/run cpu python -m pytest tests/test_*.py -q: 1082 passed, 5 skipped. New: tests/test_chem_nmr_baseline.py and four tests in tests/test_md_utils.py.
  • tools/check_skill_commands.py passes; pre-commit hooks pass.
  • Every example command was re-run from the repository root.

🤖 Generated with Claude Code

bowen-bd and others added 10 commits October 10, 2026 13:39
…itor stops

run_md treated every input as a restart because Atoms.get_velocities()
returns zeros rather than None, so relaxation and velocity initialization
were skipped and a perfect crystal stayed at 0 K. Inputs now keep their
velocities only when they carry non-zero momenta (e.g. a .traj frame).

Also:
- ExplosionMonitor reads the temperature of the integrated atoms (dyn.atoms)
  instead of the caller's copy, so it can fire.
- run_md returns status "stopped" with stop_reason when a monitor ends the
  run, as mat-md-monitors documents.
- run_md returns and writes the last MD frame as final_structure instead of
  the input structure.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Equilibration and explosion monitors on a 256-atom FCC Cu cell with
MACE-MP-small at 300 K and 100 K, validated against equipartition,
canonical temperature fluctuations and the Dulong-Petit limit.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ated examples

--baseline-correct subtracts only the minimum, which leaves a pedestal under
noisy or offset spectra and biases fractions toward 50:50. deconvolve.py and
kinetics.py gain --baseline-window/--baseline_window (median or max of a
signal-free window, applied to mixture and references alike) and
--ppm-range/--ppm_range. The noise fraction is now reported, and the
deconvolution plot scales components by area rather than peak height.

Examples: camphor reduction borneol/isoborneol ratio (Lopansri et al. 2022)
and a synthetic kinetics series validated against its reference-free rate
constant. Corrects the isoborneol reference to Lopansri et al.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…terature

Validated per signal against Fulmer et al. 2010 / Gottlieb et al. 1997 and
the published SPINUS accuracy. Corrects the SPINUS reference.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ent matching; add examples

- query_qmof.py / query_mof_db.py: --identifier now searches CSD refcodes
  (data.filename); query_qmof.py also saves band gaps and pore properties to
  qmof_properties.json.
- query_mof_db.py: a full ARC-MOF name matches exactly (ddmof_42 no longer
  returns ddmof_4291); --elements requires all listed elements by default,
  with --element-match any for mixed-metal sets; cache dir saved as ~.
- chem-db-mof SKILL.md now matches the code (Materials Cloud archive, EQeq
  charges, structure names, metadata file) and cites Burner et al. 2023.

Examples: HKUST-1 (DOTSOV01) vs experiment and the QMOF release; ARC-MOF
ddmof_559 vs Majumdar et al. 2021 and the ARC-MOF metadata.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d filters; add examples

- arXiv: multi-word queries were OR-ed (all:a b c); every term now gets its
  own field prefix and terms are AND-ed, authors are quoted phrases and
  categories are OR-ed. Errors propagate instead of returning [].
- bioRxiv: multi-word categories never matched (cell_biology vs
  "cell biology"); DOI lookup returns the latest version; medRxiv URLs use
  medrxiv.org.

Examples checked against the arXiv, bioRxiv and Crossref records. Removes
the stale mlip_search_example.json produced by the old query.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…stale TensorNet file; add examples

- MACE-MP-medium, MACE-MH-0 (3 heads), MACE-MATPES-PBE-0 O and MACE-MP-small
  Cl came from relaxations in which the molecular crystal collapsed; they
  are replaced by single points on the MP structure or converged values.
- TensorNet-MatPES-r2SCAN-v2025.1-PES regenerated with the weights the
  wrapper now loads under that name.
- get_elemental_energies.py resolves the library relative to the script,
  matches checkpoint names case-insensitively and no longer writes into
  resources/.

Examples: MP2020 corrections reproduce Materials Project corrected energies
exactly (Fe2O3, Al2O3, FeS2); library energies compared with MP GGA/r2SCAN.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…eed): shared results file

- nvalchemi: batched vs sequential static inference agrees to float32
  round-off for MACE-OMAT-0-small and TensorNet, matching the stored table.
- ml-mlip-speed: runs and --only_plot share --results_file; --hardware_name
  labels replots; PROJECT_ROOT fixed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…asure batched MD

- ml-mlip-speed: small NaCl benchmark (MACE-MP-small/medium, TensorNet,
  uma-s-1p1 up to 2000 atoms) compared with the stored GB10 reference.
- ml-mlip-nvalchemi: the batched-MD record (4.90x) does not reproduce with
  the committed locks; a quiet-GPU re-run gives 1.40x for MACE-OMAT-0-small
  at ~20 GB peak GPU memory, and TensorNet batched MD exceeded 37 GB.
  SKILL.md now reports the re-measured numbers and the memory cost.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…library

Adds the TensorNet-MatPES-PBE-v2025.1-PES alias to the 2025.2 weights, like
the other v2025.1 names, and regenerates its library file with those
weights. O stores the single point on the MP structure because the
molecular crystal collapses under relaxation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@bowen-bd bowen-bd changed the title Add validated execution examples to 11 skills and fix bugs they exposed Add validated execution examples to 12 skills and fix bugs they exposed Oct 10, 2026
bowen-bd and others added 2 commits October 10, 2026 19:30
Requiring non-zero momenta treated an explicit all-zero momenta array as
missing, so run_md relaxed the input and redrew velocities at the target
temperature, changing intentional starts from rest (e.g. NVE). The presence
of the momenta array alone now marks supplied velocities, matching
CustomMDCalc.calc. Inputs without momenta still get Maxwell-Boltzmann
velocities.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…is used

A restricted fit normalizes each reference over the fitted range, so its
signal fraction covers only the protons whose signals lie there; dividing
by whole-molecule --protons gave wrong compositions with clean WD and noise
diagnostics (e.g. 75:25 for a true 50:50). deconvolve.py and kinetics.py
take --range-protons/--range_protons; without it each --protons count is
scaled by the reference's area fraction inside the range. The camphor
example states its H2 counts (1 1) and is unchanged at 13.1 % borneol.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@bowen-bd
bowen-bd merged commit a4d3a7f into learningmatter-mit:main Oct 10, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant