fix(dspark): drain engine streams before FSDP forward AND backward - #6
Open
Boreas618 wants to merge 27 commits into
Open
fix(dspark): drain engine streams before FSDP forward AND backward#6Boreas618 wants to merge 27 commits into
Boreas618 wants to merge 27 commits into
Conversation
Boreas618
force-pushed
the
inkling-dspark-v2
branch
from
August 1, 2026 17:33
1631bbe to
81dd85d
Compare
…ion) Port of TorchSpec PR sgl-project#129 to SpecForge. Adds: - specforge/modeling/draft/dspark.py: DSparkConfig, VanillaMarkov, AcceptRatePredictor, DSparkDraftModel (subclass of DFlashDraftModel) - specforge/core/dspark.py: OnlineDSparkModel (subclass of OnlineDFlashModel) with Markov-biased logits + CE + L1 distribution distillation + confidence BCE and a pooled global-mean loss - scripts/train_dspark.py: training driver (clone of train_dflash.py) - configs/qwen3-8b-dspark.json, examples/run_qwen3_8b_dspark_online.sh - last_hidden_states surfaced from the DFlash target backends (HF + sglang) - tests/test_utils/test_dspark.py: 11 CPU unit tests
… eval) Train a dense 5-layer DSpark drafter for zai-org/GLM-5.2-FP8 (glm_moe_dsa), matching the RedHat/UCloud community consensus; validated end-to-end on a single-node tp=4 smoke against the real 753B target (loss 3.90->3.59, checkpoint save+reload). - configs/glm-5.2-dspark.json: dense 5L draft (hidden 6144, head_dim 64, block_size 7, aux target layers [1,19,38,57,76], markov 256 + confidence). - data/template.py + parse.py + preprocessing.py: glm-5.2 chat template + GLMParser (forces enable_thinking=False; loss mask = assistant content). - modeling/target/dflash_target_model.py: set_dflash_layers_to_capture + append final post-norm hidden so DSpark L1/confidence get last_hidden_states. - modeling/draft/dflash.py + core/dflash.py: optional FA4 flex path (gated off). - scripts/train_dspark.py: 4-node hardening + guarded accept-length eval hook. - scripts/eval_dspark_deepspec.py: DeepSpec mean-accepted-length eval (9 tasks). - scripts/prepare_glm52_dspark_data.py + examples/run_glm5.2_dspark_4node.sh: mixed-corpus prep + 4-node WORLD=16 DP-attention launcher.
…cept-length eval - train_dspark.py: torch.compile(dspark_model) after FSDP (SPECFORGE_COMPILE_DRAFT, on by default); flex attention + dual-source mask construction excluded via @torch.compiler.disable (they fail inductor lowering nested in an outer dynamic compile). Validated 2-GPU FSDP: ~55s warm, ~0.1s/step, dynamic shapes, finite grads. - eval_dspark_deepspec.py: rewrite the accept-length core to DeepSpec's EXACT rejection-sampling verify (logits_to_probs / sample_from_probs / sample_residual / gather_token_probs ported verbatim; accept_prob=clamp(p_t/p_d,max=1) -> rand<ap -> cumprod; +1 bonus unless EOS; accept_rate@k; per-sample seed_all(seed+idx)). Draft proposal now returns per-position draft_probs. Bit-exact vs DeepSpec-greedy at temp 0, faithful at any temperature. Default task = gsm8k; standalone defaults to DeepSpec's gsm8k protocol (500 prompts, 2048 tokens, seed 980406). Fixes a BatchEncoding.shape crash in _encode_prompt (GLM tokenizer returns a dict). - train_dspark.py: --evals-per-epoch (10 evals/epoch); in-loop accept-length eval hook now LOGS failures with full traceback (no silent swallow). - run script: num_anchors 1024 (RedHat), evals-per-epoch 10, EVAL_DATASETS_DIR default + gsm8k.jsonl build in `prepare`, SPECFORGE_COMPILE_DRAFT=1. - core/dflash.py: @torch.compiler.disable on create_dflash_block_mask; draft/dflash.py: @torch.compiler.disable on the dual-source attention (keeps block-sparse flex outside the outer compile). FA4-flex path retained (gated, off) with the SM100 findings. - .gitignore: ignore *.local.sh (local launch cheatsheets w/ credentials).
…, no IB/DeepEP) The devbox container exposes no /dev/infiniband verbs (HCAs are ACTIVE in sysfs but the char devices aren't mapped in), so NVSHMEM/DeepEP cannot init any cross-node IB transport (IBGDA fails; IBRC would too) -> Approach 1 (DP-attention+DeepEP) can't run cross-node. Add APPROACH=2: one per-node tp=NUM_GPUS engine (intra-node NVLink target, no DeepEP) + cross-node data-parallelism via the draft FSDP over NCCL (TCP when NCCL_IB_DISABLE=1). DATA_STREAMS/ACC and MEM_FRAC adjust per approach; global batch 512 preserved (A2: dp=4, acc=128). Loss-correct (objective pooled-mean is valid for the 4-replicated tp-groups). Use when IB is unavailable; prefer A1 once /dev/infiniband is exposed.
…crashes with FSDP) torch.compile(dspark_model) crashes under real varying-length data: dynamo recompiles on a new shape after a backward has populated grads and hits "assign a gradient of size [s0] to a tensor of size [s19]" (FSDP SHARD_GRAD_OP sharded-grad vs full-param) inside Qwen3RMSNorm. The 2-GPU test passed only because its shapes never forced a post-grad recompile. Eager is validated + stable and the run is target-bound, so default compile off; re-enable (=1) once the FSDP-recompile interaction is fixed.
Two DSpark training fixes surfaced by the 4-node run that showed loss=inf:
1. Resume silently loaded a RANDOM draft. Checkpoints saved while
torch.compile was enabled carry an `_orig_mod.` key prefix that neither
save_checkpoint nor the resume path stripped, so from_pretrained matched
0 keys and re-initialized the whole draft. Now:
- save_checkpoint strips both `_orig_mod.` and `draft_model.`;
- resume loads weights straight from the checkpoint safetensors (stripping
both prefixes) and RAISES if it matches nothing, instead of training a
random draft. Verified the existing epoch_0_step_2 checkpoint is 100%
finite — the inf is a forward-pass issue, not NaN weights.
2. loss=inf diagnostics + opt-in sanitizer in OnlineDSparkModel.forward /
_dspark_objective:
- [NONFINITE-INPUT] logs when the captured target hidden (context or
final) is non-finite — the likely inf source (fc(inf) -> inf logits).
- [NONFINITE-LOSS] fires only when the loss is already broken (zero cost
on the healthy path) and reports which numerator (ce/l1/conf), the
denominators, and whether draft_hidden/base_logits/last_hidden are the
source.
- SPECFORGE_SANITIZE_NONFINITE=1 (default OFF) drops non-finite tokens
from supervision and zeros their hidden so one bad sample cannot
NaN-poison all ranks via the gradient all-reduce under genuine DP.
Diagnostics default ON (SPECFORGE_DEBUG_NONFINITE=1); sanitizer default OFF
so the first run shows the raw source before we mask anything.
The GLM-5.2 launcher now defaults to the validated topology + robustness so
the real run needs no extra flags:
- APPROACH=${APPROACH:-2} (per-node tp=4 + cross-node draft DP over NVLink;
the DeepEP/IB Approach-1 path is opt-in, pending RDMA in-container).
- SPECFORGE_SANITIZE_NONFINITE=1 (was opt-in) — drop non-finite tokens from
supervision so one rare bad FP8-target-capture sample can't NaN all ranks
over a 1.5M x 10-epoch run; [NONFINITE-*]/[SANITIZE] still log the rate.
- SPECFORGE_DEBUG_NONFINITE=1, SPECFORGE_COMPILE_DRAFT=0 (already 0) made
explicit in the launcher.
All remain overridable via env (e.g. APPROACH=1, SANITIZE=0 to crash-on-bad-
sample for diagnosis). Fixed a stale trainer comment (run script sets compile
=0, not =1). The 40-step 4-node smoke was clean, so this is standing insurance.
The 4-node run hung at step ~47: rank 0 blocked in the DataLoader while ranks 1-3 spun in the target's tp=4 collective waiting for it. py-spy showed a fork()ed DataLoader worker deadlocked — mid-batch tensorize triggered an import, and while holding the import lock a GC'd wandb public-API object's weakref finalizer fired and blocked forever on api_cleanup_request (the wandb service socket doesn't exist in the fork). That worker never yielded its batch. This is exactly why the REPORT_TO=none smoke ran clean but the REPORT_TO=wandb real run hung. Fix: force num_workers=0 (load in-process, no fork, no inherited wandb finalizer). spawn workers avoid the inherit but re-exec this heavy module per worker and are fragile here; the workload is target-prefill-bound and data is pre-tokenised (Arrow mmap), so in-process loading costs ~1-2%/step. Launcher now passes --dataloader-num-workers 0 to match (the trainer forces 0 regardless).
…~2.5x) Fixes the step-46 CUDA OOM and cuts per-epoch wall clock ~2.5-3x under the Approach-2 (no-RDMA) topology. Recipe untouched: num_anchors stays 1024, global batch stays 512, optimizer steps/epoch stay 1445, same LR schedule. 1. Chunked DSpark objective (SPECFORGE_OBJECTIVE_CHUNK_BLOCKS=128, 0=legacy). The objective materialized a [nb,7,154880] f32 logits/probs stack — 24.91 GiB measured at nb=1024 — inside the ~45 GB left beside the sglang static pool; backward pushed it over. Now the block dim is processed in slices under torch.utils.checkpoint (recompute-in-backward): peak 3.93 GiB, +46 ms on the worst-case sample. Validated exactly grad-equivalent to the legacy path on CPU (loss, all metrics, draft/markov/confidence grads) and loss-delta 2.8e-5 in bf16 at real dims. Includes a cuBLAS fix: vocab linears on sliced 4D tensors hit a degenerate batched-GEMM kernel (37.8 ms vs 1.1 ms flattened) — all vocab GEMMs now flatten to 2D first. 2. TP-batch scatter (SPECFORGE_TP_BATCH_SCATTER=1, tp-replicated target only). Without DP-attention every rank in the node's tp group holds the same batch and identical target hiddens; training the draft on all of it just computes tp_size identical gradient copies. Now the node batch (default 4 under APPROACH=2) is prefilled once by the tp=4 target, then each rank trains the draft on a distinct 1/tp slice trimmed to its true length -> 16 unique data streams, draft compute deduplicated, ACC 128->32. 3. generate_dflash_data strips right-padding per request (the 750B target no longer prefills pad tokens at batch>1) and re-pads the returned hiddens; pad positions are never read (loss_mask=0, blocks attend <= anchor). 4. Launcher: BATCH_SIZE defaults to 4 under APPROACH=2, plus PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True (the OOM report showed 12.8 GB reserved-but-unallocated fragmentation; safe — the in-process engine runs disable_cuda_graph=True). Validated end-to-end: single-node 4-GPU smoke (batch 4, scatter, chunked) ran 40 steps clean — no OOM/non-finite, losses match the pre-change runs, ~1.1 s/iter for 4 samples vs ~0.7 s/iter for 1 before (~2.5x throughput); batched prefill confirmed pad-stripped (hs_shape 5748 tokens for 4 samples).
anchor sampler, tau_probabilistic Full training-flow audit against deepseek-ai/DeepSpec@005e03b. Already exact (verified line-by-line against their source): label convention (slot k -> anchor+k+1, all slots supervised), eval mask (cumprod prefix), noise embed (slot0 = anchor token), position ids, attention mask (strict kv < anchor, bidirectional intra-block), backbone (hidden_norm(fc(.)), k_norm over ctx+noise concat, RoPE cos[-q_len:]), VanillaMarkov bias on teacher-forced prev ids applied before CE AND L1, confidence BCE on [hidden, prev_emb] vs detached accept_rate, loss 0.1*CE + 0.9*L1 + 1.0*conf with exp(-k/4) decay and pooled global-mean reduction (their 3 all-reduced denominators are all loss_weight_mask.sum() == our shared denominator), frozen target embed+head, BF16Optimizer/AdamW wd=0/cosine+warmup 0.04/ lr 6e-4/bf16/epochs 10/global batch 512/max_len 4096/seed 42. Three mismatches found and fixed: 1. Grad clipping was LOCAL-SHARD norm (BF16Optimizer clips fp32 master shards) vs DeepSpec's global-norm FSDP.clip_grad_norm_ — under 16-way sharding that is a ~sqrt(world)x looser, non-uniform clip. Now clip the FSDP model by the global norm before optimizer.step() (DeepSpec order); the optimizer's internal local clip provably no-ops after that. 2. Anchor sampling deviated 3 ways from DeepSpec sample_anchor_positions: no first-target-valid rule (loss_mask[p+1]) -> anchors wasted on zero-supervision blocks at turn boundaries; candidates capped at seq_len - block_size (DeepSpec allows to seq_len-2 with eval-mask truncation); off-by-one in the keep cap (batch_max_valid - 1). OnlineDSparkModel now overrides the DFlash sampler with a DeepSpec-exact port — validated bit-identical anchors + keep masks vs DeepSpec's own function under shared RNG across 200 randomized trials. One disclosed deviation kept: anchor tensor width caps at the batch max keep count instead of always padding to num_anchors (identical supervision; dummy columns only burn draft-MLP compute). 3. Added DeepSpec's tau_probabilistic train metric (expected accepted drafts per block + 1, over valid blocks) to both objective paths; chunked-vs-legacy equivalence re-verified incl. the new metric. Intentional deviations (owner-locked / infra): num_anchors 1024 (RedHat) vs DeepSpec reference 512; online sglang FP8 teacher vs offline bf16 HF cache (same tensors: layer-output hooks + post-norm final hidden); shard_grad_op vs no_shard; no no_sync during accumulation (math-identical, ~4% comm); compile off (FSDP-recompile crash); chunked objective and TP-batch scatter (both proven equivalence-preserving).
…odel rescue) The reshard-safe resume resets Adam moments; the first optimizer steps are then sign-like kicks of ~lr per coordinate (m_hat/sqrt(v_hat) = +-1 at t=1). On a converged model at full LR this is destructive — observed live: rollback to a healthy checkpoint (tau 4.06) degraded to tau ~1.1 within ~5 optimizer steps of resuming at lr 5.76e-4. Fix: linearly re-warm the LR over the first N optimizer steps after a moment-reset resume (SPECFORGE_RESUME_LR_REWARM_OPT_STEPS, default 64). The second moments re-estimate on real gradients while the weights barely move; identical factor on every rank keeps sharded updates consistent. Implementation note: the damp must be applied inside BF16Optimizer.step() (new optional lr_scale arg) and the original lr restored BEFORE scheduler.step() — torch schedulers like CosineAnnealingLR are recurrences on param_group["lr"], so a damped value left in place permanently drags the remaining schedule down (validated: naive one-shot damp collapsed the schedule 1e-3 -> 9.4e-5; the fixed path keeps the schedule exact while the effective LR ramps 0.25x -> 1x).
edge-of-stability collapses The converged drafter (tau ~4+, gsm8k accept_len 4.76) collapsed twice at the cosine schedule's peak region (5.76e-4): ce 1.8 -> 40+ with onset within ~3 optimizer steps of running at full schedule LR, at two different data positions (micro steps ~75.9k and ~77.1k), and was stable through the entire damped post-resume re-warm both times. Classic sharp-minimum instability; the cosine stays above half-peak until ~55% of training, so waiting it out is not an option. SPECFORGE_LR_SCALE (default 1.0 = DeepSpec-parity recipe; GLM launcher sets 0.5) multiplies the effective LR while leaving the schedule shape exact — applied through optimizer.step(lr_scale=...), which restores the group lr before the recurrent scheduler advances. Composes with the post-resume re-warm ramp. A one-time cross-rank all-reduce check aborts startup if ranks disagree on the scale (a silent mismatch would diverge the sharded updates).
Distinguishes gradient explosion (grad_norm >> max_norm at collapse onset) from curvature/step-size instability (collapse with flat grad_norm — Adam's update magnitude is ~lr per coordinate regardless of gradient scale, which clipping cannot bound).
(DeepSpec-style incremental verify on the sglang bridge) The DeepSpec eval re-prefilled the whole growing sequence every verify step (the training bridge only exposed stateless prefill) -> O(n^2) target compute per sample, ~50-100x DeepSpec's KV-cached incremental verify (DynamicCache + crop). New: start_cache_session()/end_cache_session() give the bridge a persistent RadixCache; each generate_dflash_data call prefix-matches against it (init_next_round_input), computes only the unmatched suffix, and inserts the COMMITTED tokens back (cache_finished_req with cache_commit_len). Only committed tokens are ever cached and drafted-block KV is freed, so a future match can never overrun the positions the verifier reads. Hidden states are returned suffix-only with a prefix_len offset (DFlashTargetOutput.prefix_len); outputs split by extend_input_len (identical to the old full-length split when no session is active). Training path untouched. Two traps fixed along the way: - Paged-allocator double free: free() frees WHOLE pages (unique(idx // page)); cache_finished_req already frees the page holding the unaligned commit boundary, so the manual tail free must start at the next page boundary. - eval cmd: multi-node dp x tp sweep entry (cmd_eval) in the 4-node launcher. Eval loop: unified relative indexing (prefix_len=0 without a session), context maintained incrementally, session released in a finally wrapper, SPECFORGE_EVAL_KV_REUSE=0 forces the legacy path. Validated: 4-rank lockstep breadcrumbs across cached+control phases, accept-length parity with the legacy path (4.88 vs 5.08 on n=4, within greedy near-tie noise from different kernel shapes), and per-64-page prefix matching observed live. Rows now flush immediately and partial JSON is checkpointed after each task.
… eval default
Root cause of the ~50% training-data loss: the glm-5.2 chat template hardcoded
assistant_header="<|assistant|><think></think>" (the EMPTY think block). GLM
renders a thinking-ON assistant turn as "<|assistant|><think>{reasoning}</think>
{answer}", so the literal-empty-block header never matched -> those turns got a
zero loss mask and were dropped by the min_loss_tokens filter. That is exactly
the 1,498,000 -> 739,583 (49.4%) filter seen in training: 100% of the dropped
half were thinking-ON regens (the mgoin open-perfectblend regenerations). The
drafter therefore trained thinking-OFF-only and was evaluated thinking-OFF.
Owner decision: target GLM-5.2's default THINKING-ON mode; supervise reasoning
+ answer. Fixes:
- template.py: assistant_header -> "<|assistant|><think>" (the always-scaffolding
prefix) + assistant_pattern_type="glm".
- parse.py: new "glm" assistant_pattern
<\|assistant\|><think>(?:</think>)?([\s\S]*?(?:<\|user\|>|$))
-> supervises {reasoning}</think>{answer} for thinking-ON and just {answer}
for thinking-OFF (the empty-block </think> scaffold is skipped, unsupervised).
Validated on CPU: retention 49.4% -> 99.8%; thinking-ON span starts at the
reasoning and includes the generated </think>; thinking-OFF span excludes the
scaffold </think>.
- train_dspark.py: (1) hard guard — abort if <90% of samples survive the
loss-mask filter (SPECFORGE_MIN_RETENTION), so a template/mask mismatch can
never again silently halve the corpus; (2) cache key versioned
(maskv2-glm-thinkhybrid) — the processed-dataset cache keyed on template NAME
would otherwise reuse the stale 49.4%-masked tensors after this content change;
(3) in-loop --eval-max-new-tokens default 256 -> 1024 so thinking-ON reasoning
isn't fully truncated.
- eval: SPECFORGE_EVAL_ENABLE_THINKING default OFF -> ON (matches deployment +
training target); launcher eval max_new 1024 -> 2048 (room for the reasoning
chain); EVAL_TAG records the thinking mode.
Recipe implication (training deferred): retention ~2x -> ~1.5M samples/epoch,
so steps/epoch and the cosine schedule length ~double vs the prior run. Stale
processed_dataset cache cleared.
… fixed The training path accreted ~10 SPECFORGE_* env knobs, two import-time monkeypatches, a broken resume branch, and one-off debug prints during bring-up. The flow is now settled, so most knobs have a single correct value. Net: the training path exposes ONE env var (SPECFORGE_DATA_NUM_PROC) down from ~10, with zero change to training numerics (verified: chunked==legacy objective loss+grads bit-parity; retention still 100%; defaults equal the old env values). scripts/train_dspark.py - Delete the torch.compile path (SPECFORGE_COMPILE_DRAFT) + its per-rank inductor/triton cache monkeypatch — compile crashes with FSDP on the data-dependent num_anchors shape; eager is the validated path. - Delete SPECFORGE_OFFLOAD_MASTER (pass offload_master=False) and the SPECFORGE_RESUME_FULL_OPTIM branch (impossible multi-rank — only rank-0's optim shard is saved; reshard-safe is the sole correct path). - Hardcode FSDP shard_grad_op (SPECFORGE_FSDP_STRATEGY) and the retention guard (MIN_RETENTION=0.9 module constant). - Promote the two genuine recipe hyperparameters to real CLI args: --lr-scale (0.5) and --resume-lr-rewarm-steps (64), keeping the cross-rank consistency check. - De-magic the PG-timeout monkeypatch (PG_TIMEOUT_MINUTES=45, now setdefault so it no longer silently clobbers callers) and delete the dead --dist-timeout arg it had been overriding. - TP-batch scatter: drop the env gate; auto-enable from topology (tp-replicated + batch divisible by tp). - Resume-path raw print() -> print_on_rank0 (stop duplicating across ranks). specforge/core/dspark.py - Convert the three objective env knobs to OnlineDSparkModel constructor params: objective_chunk_blocks=128, sanitize_nonfinite=True, debug_nonfinite=True. Legacy _full_numerators stays reachable via objective_chunk_blocks=0 (kept as the equivalence reference). Drop the now-unused `import os`. specforge/modeling/target/dflash_target_model.py - Remove the [capture] and one-shot [DEBUG _extend] scaffolding prints. Keep the ServerArgs-drop info line, the [KV] debug (gated), and the sglang serial-load / moe-runner-backend escape hatches (shared by all entrypoints). examples/run_glm5.2_dspark_4node.sh - Drop the exports for the removed/hardcoded knobs + redundant unsets; pass --lr-scale / --resume-lr-rewarm-steps in the torchrun call; drop --dist-timeout. Deliberately untouched (blast radius): optimizer.py (shared lr_scale default used by 7 other trainers), train_dspark_v4/moe/dflash (own knob copies), the shared sglang-load knobs, the FA4 flex gate in shared core/dflash.py, and Approach-1.
…pproved) Force the DeepSpec recipe values where they can apply to the GLM-5.2 target, per owner decision (with the known trade-offs accepted): Training - num_anchors 1024 -> 512. DeepSpec canonical (all their dspark configs use 512). Halves supervision density/step; also the likely reason DeepSpec is stable at unscaled 6e-4 (our earlier collapse was at 1024). - Effective LR 0.5x -> 1.0x (unscaled 6e-4) = DeepSpec. --lr-scale default is now 1.0 (launcher + argparse); the 0.5 escape hatch remains (set LR_SCALE=0.5) if the run re-collapses at the schedule peak. grad-norm telemetry + sanitizer stay as the safety net. Evaluation (DeepSpec-exact protocol) - temperature 0.0 -> 1.0 (DeepSpec's stochastic rejection-sampling default). - per-task caps: EVAL_LIMIT 128 -> 0, which now maps bare --tasks names to the DeepSpec DEFAULT_TASKS caps (gsm8k 500, math500 500, aime25 30, humaneval 164, mbpp 256, livecodebench 500, mt-bench 80, alpaca 500, arena-hard-v2 500) via a fallback added to _normalize_tasks. max_new 2048 + seed 980406 already matched. Already identical (verified earlier, unchanged): objective/loss math, block_size 7, 5 layers, markov 256 + vanilla, confidence + with_markov, α 0.1/0.9/1.0, decay 4.0, base lr 6e-4, warmup 0.04, wd 0, bf16, global_batch 512, epochs 10, grad-clip 1.0, max_len 4096. NOT changed (target-dependent, cannot be identical to a Qwen3/Gemma config): target=GLM-5.2-FP8, target_layer_ids [1,19,38,57,76] (78 layers), mask_token 154856, chat_template glm-5.2, vocab/hidden. NOT forced (owner left unchecked; would reintroduce known failures): torch_compile (FSDP crash), dataloader num_workers=4 (wandb+fork deadlock). Kept for safety (non-numerics): FSDP shard_grad_op (vs no_shard), save-interval 500 (vs 3000) for rollback granularity.
…igned in-loop eval
Fresh-run reset of the GLM-5.2 DSpark pipeline:
- Data: corpus = mgoin/open-perfectblend-glm5.2-regen ONLY (codealpaca mix
removed everywhere); whole corpus trains, no eval carve (the dedicated
DeepSpec accept-length suite is the eval; --eval-data-path was dead code
in this flow and is no longer passed).
- Thinking-ON only: GLMParser renders enable_thinking=True (training prompt
= deployment prompt byte-for-byte, incl. the "Reasoning Effort: Max"
header, outside the loss mask); eval _encode_prompt has no thinking-OFF
path; SPECFORGE_EVAL_ENABLE_THINKING removed. Cache key bumped to
maskv3-glm-thinkon (train + launcher warm cache kept in sync).
Validated on 500 real regen rows: retention 100%, header excluded from
the mask, span = {reasoning}</think>{answer}.
- In-loop eval: gsm8k + mbpp + mt-bench @ 32 prompts each (--eval-tasks),
temp 1.0 / max_new 2048 / seed 980406 == the final-eval protocol; the
in-loop subsets are seed-matched prefixes of the final runs. prepare now
builds all three jsonls per node (byte-identical to the reference files;
a jsonl present on some replicas but not others deadlocks the eval's dp
all-reduce).
- Launcher: Approach 2 only (per-node tp engine + cross-node FSDP DP);
DP-attention/DeepEP path removed (no /dev/infiniband in this container).
MEM_FRAC default 0.78.
- Trainer defaults resolved to the DeepSpec/owner recipe: block 7, 5 draft
layers, 10 epochs, max_length 4096.
- Docs: REGEN_OPEN_PERFECTBLEND.md regen+verification guide; execution doc
updated to match all of the above.
…-level backbone Ports the DSpark trainer as evolved for a second target family (Inkling, 42-layer hybrid-SWA MoE, NVFP4, eos-less tokenizer): - target: frozen embed/head allocated at padded_vocab_size (201024), not vocab_size; embed/head key remapping per target arch - data: assistant-pattern + ThinkingParser support, nda_inkling_thinking chat template, seeded node-file sampler + content-hash dedup in prepare, tokenized-cache key includes train-data content hash (prepare and train derive identical keys) - eval (in-loop): stop-token resolution falls back to the draft config's eos_token_id list when the tokenizer defines no eos (raises if neither exists); task list auto-discovers <task>.jsonl in eval-datasets-dir - resume: world-consistent resume guard — all ranks agree on the newest checkpoint present on every node, falling back to the newest common one (fixes a deterministic post-checkpoint rendezvous deadlock) - backbone v2 is config-level: KV injection + block-local attention already in the modeling stack; uniform target taps, target-matched GQA geometry, and block-size-adaptive loss decay are all expressed in the draft config - capture-parity probe (probe_nda_inkling_capture.py): scale-aware P1 + stream-distinctness P2b checks before first smoke Deliberately NOT included (companion PRs): native-list tools fix (fix/tools-native-list, based on main), evals-off guard, engine-stream drains, dual-rope-schema save. Excluded by owner selection: --fixed-lr, slim checkpoints, dual-node saves, watchdog machinery.
The colocated in-process SGLang engine's pool-reset kernels race FSDP collective launches, wedging NCCL with the signature: default_pg all_reduce completed + _all_gather_base scheduled-never-launched on a rank subset. Draining (torch.cuda.synchronize) before the draft forward covers the capture path; a second drain immediately before .backward() covers pool-reset work queued during the forward — without it the reset kernels race BACKWARD_PRE all-gathers. Observed wedge cadence dropped ~5x (30-45 min -> 2-4 h) with the backward drain in place.
Boreas618
force-pushed
the
fix/engine-stream-drains
branch
from
August 1, 2026 17:34
88b4920 to
c2d77a9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Item 8. The colocated in-process SGLang engine's pool-reset kernels race FSDP collective launches: signature is
all_reduce completed+_all_gather_base scheduled-never-launched on a rank subset. The pre-forward drain covers the capture path; the second drain immediately before.backward()covers pool-reset work queued during forward, which otherwise races BACKWARD_PRE all-gathers. Wedge cadence dropped ~5× (30-45 min → 2-4 h) when the backward drain landed, measured across two multi-day 2-node runs.Stacked on #inkling-dspark-v2.
🤖 Generated with Claude Code