Width ablation at full-MP scale (W96/W128/W160) + CoreWeave GB200 training infra - #160

Draft

forklady42 wants to merge 60 commits into

mainfrom

betsy/coreweave-w96

Commits on May 28, 2026

Add large-scale GGA + GGA+U fp32 training config

forklady42
and
claude
committed
b79d6cb
Add Modal training + Globus data-load for large-scale GGA/GGA+U run

forklady42
and
claude
committed
2e2ce78
Add prep_volume helper to wire the data Volume after Globus transfer

forklady42
and
claude
committed
0a1e0f6

Commits on Jun 15, 2026

Point populate_volume at oa-electrai with dedicated read-only secret

forklady42
and
claude
committed
c841b35

Commits on Jun 17, 2026

Parallelize populate_volume S3 download with ThreadPoolExecutor

forklady42
and
claude
committed
35160da
Use .spawn() so the populate_volume pull survives local CLI disconnect

forklady42
and
claude
committed
f279059

Commits on Jun 18, 2026

Pack zarr stores as .zarr.zip to fit under Modal Volume's inode cap

forklady42
and
claude
committed
cf2e7fd
Use .spawn() in train.py so multi-day runs survive local CLI disconnects

forklady42
and
claude
committed
1b65484
Add Lambda Cloud runbook (scripts/lambda/) for the long-run training campaign

forklady42
and
claude
committed
bee89bc

Commits on Jun 19, 2026

Source ~/.bashrc in lambda run_training.sh for non-interactive callers

forklady42
and
claude
committed
fea5263
Lambda: prep_data accepts both .zarr.zip and .zarr/ (S3 is unpacked)

forklady42
and
claude
committed
6662ed6
Lambda run_training: grep WANDB_API_KEY from ~/.bashrc, not source it

forklady42
and
claude
committed
42ef893
Lambda: default data + checkpoints to NFS filesystem; fix run_training

forklady42
and
claude
committed
1924e0e
Lambda run_training: simplify /data/ -> DATA_ROOT sed (YAML list-dash safe)

forklady42
and
claude
committed
3ffae7f
Lambda: auto-detect NFS_ROOT from /lambda/nfs/*

forklady42
and
claude
committed
b4e16e0

Commits on Jun 20, 2026

Lambda run_training: WANDB_MODE_OVERRIDE env var for offline/disabled modes

forklady42
and
claude
committed
c094da0
Lambda run_training: anchor path-rewrite sed on leading space

forklady42
and
claude
committed
fafb857

Commits on Jun 22, 2026

Lambda multinode: drop-in 2-node DDP launcher + runbook

forklady42
and
claude
committed
81c6ea0
Add PORT_PLAN.md: migration plan from H100:4 -> 2-node H100:16

forklady42
and
claude
committed
de3edf7
Add MONITOR.md + monitor_status.sh: monitoring script + LLM-driven hourly cron template

forklady42
and
claude
committed
5436fcd
Lambda monitor: EC2 resident Claude agent (active operator)

forklady42
and
claude
committed
483b306
Lambda monitor: default to Opus (claude-opus-4-8) for monitoring

forklady42
and
claude
committed
af2b108
Lambda monitor: escalation degrades to a journal banner when no Slack webhook

forklady42
and
claude
committed
f1aa848
Lambda monitor: fix snapshot cmd allowlist match; wire Slack state via CONTEXT

forklady42
and
claude
committed
8f44ba0

Commits on Jun 24, 2026

Add width-64 GGA+GGA+U config + w64 mode for Lambda training

forklady42
and
claude
committed
8a34d49
W64 run: grid-size cap + DDP timeout + frequent resume checkpoint

forklady42
and
claude
committed
7c7fc95

Commits on Jul 13, 2026

W64: switch precision fp16-mixed -> bf16-mixed (fp16 diverged to NaN)

forklady42
and
claude
committed
3c70889
Add train_watchdog.sh: cron-driven NaN/stall/down alert + auto-stop

forklady42
and
claude
committed
2f79873

Commits on Jul 17, 2026

Use PyTorch cu128 index for CUDA torch wheels on linux-aarch64

forklady42
and
claude
committed
bb249dc
Add width-96 config for the width ablation (single-variable vs W64)

forklady42
and
claude
committed
fe8342b
Add CoreWeave stage-in script for charge-density data

forklady42
and
claude
committed
c6e9df1
Merge branch 'betsy/gga-ggau-w96' into betsy/coreweave-w96
forklady42
committed
8afaa9f
Point W96 config at the CoreWeave stage-in layout

forklady42
and
claude
committed
7e8b068
Add CoreWeave training wrapper and dry-run script

forklady42
and
claude
committed
d7d4c69

Commits on Jul 18, 2026

Switch staging and checkpoint sync from s5cmd to rclone

forklady42
and
claude
committed
59051b0
Stage into /uv/cache: the only host-persistent mount in Iris task pods

forklady42
and
claude
committed
e04fc55
Run wandb offline with a sync sidecar on CoreWeave

forklady42
and
claude
committed
4b85565

Commits on Jul 19, 2026

Use step-based interval for the resume checkpoint, not wall-clock

forklady42
and
claude
committed
616991a

Commits on Jul 20, 2026

Promote the newest last*.ckpt before resume

forklady42
and
claude
committed
179935a
Add W128 config and generic Iris submit helper

forklady42
and
claude
committed
8db696e

Commits on Jul 21, 2026

Add exploratory W256 config for dry-run evaluation

forklady42
and
claude
committed
6966ae8
dry_run.py: grid ladder, multi-iter timing, cudnn-benchmark probe flags

forklady42
and
claude
committed
40e4a4f
Add W192 config: next width rung if it clears the cuDNN kernel cliff

forklady42
and
claude
committed
5d3bb7c
Add voxel-band counter for width-specific kernel-cliff caps

forklady42
and
claude
committed
333c1a2

Commits on Jul 22, 2026

dry_run.py: add --n-channels override for width-ceiling probes

forklady42
and
claude
committed
37ff4a0
Add W160 config: widest cliff-safe rung on the unmodified capped dataset

forklady42
and
claude
committed
63325c0

Commits on Jul 28, 2026

submit.sh: raise default retry budget for preemption-heavy periods

forklady42
and
claude
committed
fb6f51c

Commits on Jul 29, 2026

Name wandb runs from cfg.run_name

forklady42
and
claude
committed
07c80d7
Shorten wandb run names to w{width}_{MMDD-HHMM}

forklady42
and
claude
committed
1222a42

Commits on Jul 30, 2026

Add LR/weight-decay-vs-width sweep: stage-A configs, split subsampler, AdamW option

forklady42
and
claude
committed
bbb924b

Commits on Jul 31, 2026

Sweep stage A: W64 lr=8e-3 edge extension + W&B review script

forklady42
and
claude
committed
a91a49f
Sweep stage B: WD grid at per-width lr* (32=2e-3, 64=4e-3, 96=2e-3)

forklady42
and
claude
committed
0a678da
review_lrwd_sweep: keep _rep2 noise-bar reruns separate from stage-A twins

forklady42
and
claude
committed
b10cf57

Commits on Aug 1, 2026

Sweep doc: record stage A+B results and stage-C recipes

forklady42
and
claude
committed
b2a9826
Stage C confirmations (W32/64/96 tuned lr) + W128 at lr 2e-3

forklady42
and
claude
committed
d69acba
Reconcile resume checkpoints against the bucket with --update

forklady42
and
claude
committed
a04d07c

Commits on Aug 4, 2026

Fix ckpt_path in stage-C/W128 tuned configs: match run_training.sh stem strip

forklady42
and
claude
committed
9f0a387

Commits on Aug 5, 2026

Refuse resume when local last.ckpt is staler than the bucket's

forklady42
and
claude
committed
946c43d

Commits on Aug 7, 2026

W128 withheld-test-set eval: 0.481% combined NMAE (GGA 0.480 / GGA+U 0.483)

forklady42
and
claude
committed
b142576

Commits on Aug 31, 2026

Add final width-ablation figures

forklady42
and
claude
committed
aafadb5