[Stepping] Relocate the enablement subsystem and make ENABLEMENT the sixth phase

[ZhengGong-amd]

Description: what and why

Three related changes to the enablement subsystem, landed together because each depends on what the previous one establishes.

Part 1 — module relocation (commits 1–8). Enablement code lived in six places across agents/ and orchestrator/. This collapses it to three owners, deletes the repo's only reverse dependency edge, and moves 26 test files next to the code they exercise.

The reverse edge is the substantive part: agents/framework/enablement_ops.py reached back into orchestrator.framework.paths through a function-local import wrapped in a bare except Exception: pass that existed only to hide the cycle. As orchestrator/enablement/mandate.py that becomes an ordinary relative import and the guard disappears. Likewise classify_failure had no consumer inside the agent runtime — all 22 call sites were in the orchestrator — so living in agents/ made the orchestrator import agent Python in-process and rendered the subprocess JSON contract decorative. It moves to common/failure_signature.py.

Two silent failures fixed in passing: build_lifecycle._driver_command spawned python -m against a hardcoded module path (now derived from targeted_build.__name__), and test_git_foreign_checkout_callers was scanning a 33-line shim instead of the module it meant to guard.

Part 2 — ENABLEMENT as the sixth phase (commits 9–17). The chain is now:

PRELUDE → ENABLEMENT → FRAMEWORK_AGENT → KERNEL_AGENT → SWEEP → CLOSE

ENABLEMENT is entered from PRELUDE when the lane is admitted and baseline_failure_streak >= 1. A run whose baseline boots first try never enters it — streak == 0, so PRELUDE goes straight to _post_prelude_target().

compute_next_phase gains two keyword arguments (enablement_enabled, enablement_in_flight) supplied by the already-async _advance_phase_if_needed. machine_state.py gains no new first-party imports; the facts arrive through the same channel kernel_enabled and optimize_enabled already use.

The normal exit requires three conjuncts: baseline_tput > 0, not validation_pending, and no enablement work still in flight. The third is what makes the close-guard deletion sound — _maybe_rearm_authored_lane routes on the result's lane, not the current phase, so a build outliving its round would otherwise reopen validation_pending from inside FRAMEWORK_AGENT.

This is a net reduction in mechanism. enablement_close_guard_active(), MAX_SKIP_TO_CLOSE_SUPPRESSIONS, the skip_to_close_suppressions field, the intent-router suppression block, and the dead "enablement_stalled" stop reason are all deleted; the phase boundary now expresses what they approximated.

Budget: FRAMEWORK_AGENT 40→38%, KERNEL_AGENT 50→47%, and ENABLEMENT deliberately carries no share (see Part 3).

Follow-ups (commits 13–17) close what a branch-wide sweep turned up: a test left broken by a method rename, zero coverage on the new phase predicate, and EnablementBreakdown reaching the JSON but never the Markdown report.

Part 3 — let the phase finish a bring-up it has already achieved (commits 18–28). Part 2 gave enablement a phase boundary; running it against a real model showed the phase could not cross that boundary. A GLM-5.3 MXFP4 session spent its whole budget in ENABLEMENT after its patch had worked: the patched server loaded 409 GB of weights, captured graphs, served 936 tok/s and scored gsm8k 0.975, while the phase recorded baseline_tput = 0, rated the round "advanced", and kept authoring patches against a wall it had cleared. Four defects had to line up. None is a missing mechanism; each is an existing one applied at the wrong scope, so every fix removes a special case rather than adding a guard.

Runnability was decided by a log scan that a chatty log defeats. runnable_decision asked the bring-up ladder whether the server served, which the ladder cannot observe: it matches readiness substrings in a capped window of server.log. A build whose kernel layer logs per shape pushed application startup complete to byte 329,448 of a 4.9 MB log, so a fully served boot classified as stage_reached=ENGINE_INIT — indistinguishable from a hang. The comment defending the choice said throughput "cannot separate a slow server from a dead one", which confuses a measurement's magnitude with its existence: is_valid_measurement already requires a positive throughput and a completed request, and a dead server completes none. The gate now takes the served witness from the measurement and leaves the ladder the job it can do — round_advanced and the failure it explains. boot_timed_out goes with it: its evidence key was never written, so the branch was unreachable from its only caller. The log reader is still fixed, for the reason that survives — an understated stage_reached makes two unequal boots compare as equal.

Only one of the two origins could establish a baseline. baseline_tput has one writer, reachable only from a promoted baseline task. An eval-origin KEEP opened the revalidation window that produces that task; a boot-origin KEEP set succeeded outright and dispatched nothing. Since a specialist cannot run a baseline and integrate_patch deliberately keeps its own measurement out of the anchor, boot-origin had no actor left that could satisfy its own exit condition. Both origins now open the window, and the promote path is the only place that sets succeeded. It honours --no-eval there, since accuracy_meets_floor demands a positive score such a session never produces.

baseline was advertised to the model and then refused. The phase table's comment said baseline is Coordinator-internal here, but named a global set to enforce it — which cannot carry the rule, because baseline is the one action PRELUDE exists to propose. So the phase offered it, the decision framework told the model to propose one whenever baseline_tput == 0 (always, in this phase), and the round guard denied every attempt. PHASE_COORDINATOR_RESERVED now names what a single phase keeps for the Coordinator, allowed_actions_for subtracts it, and the prompt reads the same table. baseline stays in the phase's allowed set so the revalidation survives the transition sweep; what changes is who may ask for one.

The phase carried an optimisation budget. A budget divides the session between phases that produce a result; ENABLEMENT exists because the combo produces none. The 0.05 share was already inert — compute_next_phase never called phase_cap_exceeded for this phase — so the row is dropped rather than the check wired up. phase_cap_seconds already separates an absent key (no cap) from a zero share (no wall clock), and the renderer already prints cap=unlimited. The phase keeps the bounds that suit it: the consecutive-stall cap on rounds, and the session clock. --max-minutes-enablement-pct goes with the row rather than staying as a flag that parses a cap nothing enforces (caught in review).

Found alongside, in the same area: IR-1's stale-process scan excluded only os.getpid(), so the launcher shell — whose argv quotes the whole command — matched its own patterns and failed the gate on an idle machine. Three executors are module-level singletons built before the CLI pins the session, so resolving session_dir in __init__ froze them on the workspace root and every session on a shared WekaFS workspace wrote its bring-up observations into one directory; it is resolved on read now, owned by one descriptor beside the resolver. And a bare --resume-from rebuilt --max-hours and the stop target from the flags, so it shortened a 24 h session to the 2 h default and closed the leg as time_exhausted before its first action, and dropped the objective — which is what robustness_monitor.sh does on every auto-resume, since it passes no flags at all.

Two unrelated fixes, done in passing (commits 29–30). The argv preflight probed through parse_known_args, which vLLM's parser does not use to expand --<group>-config.<field>, so every dotted flag read as unrecognised and preflight spent its one repair dropping flags the server would have taken; one of them bounded the profiler, and a roofline recorded 25.7 GB of trace over the whole workload instead of a steady-state window. Separately, the robustness monitor treated the presence of reports/final.* as terminal, but the crash path writes one as a safety net and a resume clears stop_reason without removing it, so the monitor read a running session as finished and abandoned it — state.json decides now, with the artifacts standing in only when there is no state to read.

Linked issue(s): close/fix refs

None.

Tests: added/updated? commands run?

Both.

Added: five cases covering the ENABLEMENT entry and exit predicate (neither enablement_entered nor enablement_done was exercised by anything, and enablement_enabled=True had never been passed); test_enablement_relocation_completeness.py, a git ls-files sweep with a self-validating allowlist, an importability check for all ten canonical module paths, and a spawn-path check for _driver_command. For Part 3: test_bringup_log_reader.py (a served boot whose readiness marker lands past the head window still reads as booted, and a tail wall still outranks a middle milestone); test_resume_budget_objective.py (six cases over the restore, including that an explicit flag wins and that one named target replaces the persisted objective rather than joining it); the phase-reservation and admission-gate cases; the ladder-independence case on runnable_decision; and the eval_disabled promote path.

Fixed: test_environment_fault_round.py bound a lane helper by its pre-rename name, raising AttributeError at collection and silently dropping all 13 of its tests.

Updated: constant assertions (PHASE_NAMES, budget identity/sum, _PHASE_ORIENTATION, CLI redistribution figures, PHASE_GOAL_BLOCKS); three three-strike tests now set enablement_mode="off" to stay honest about the fast-fail path they assert. Six close-guard and suppression-counter tests deleted along with the mechanism. In Part 3, the lane and breakdown tests that asserted a boot-origin KEEP sets succeeded synchronously now assert the revalidation window it opens, the eval-origin-only case is folded into the general one rather than kept beside it, and the enablement gate's bench stubs carry completed_requests — which is what the real VariantResult has always reported.

pytest src/hyperloom/orchestrator/enablement/ src/hyperloom/orchestrator/bringup/ \
       src/hyperloom/orchestrator/actions/executors/tests/ \
       src/hyperloom/common/tests/test_failure_signature.py \
       src/hyperloom/inference_optimizer/tests/test_phase_state_machine.py \
       src/hyperloom/inference_optimizer/tests/test_machine_state.py \
       src/hyperloom/inference_optimizer/tests/test_policy_gate.py \
       src/hyperloom/inference_optimizer/tests/test_resume_budget_objective.py \
       src/hyperloom/inference_optimizer/tests/test_breakdown_report_integrity.py \
       src/hyperloom/inference_optimizer/tests/test_environment_fault_round.py
ruff check . && ruff format --check .

Passing on the touched surfaces (848 + 270 on the two largest selections); leaving the full suite to CI.

Breaking changes: yes/no (details if yes)

No, for anything outside this repo. Internally, resume across the change is not supported: a session whose state.json records phase: "ENABLEMENT" cannot be resumed on pre-merge code. New sessions are unaffected, and EnablementRound.from_dict filters unknown keys, so older state.json files load without migration.

One in-repo behaviour change worth naming: a boot-origin enablement KEEP no longer sets succeeded at the moment it lands. It opens a revalidation window and succeeded follows when that baseline promotes, which is what eval-origin already did. The lane event stays open for that interval rather than closing at the KEEP.

PR addresses single concern: yes/no (details if no)

No — three, plus two unrelated fixes. The phase insertion needs the relocated modules to avoid re-creating the import cycle it deletes, and Part 3 is what running the phase from Part 2 against a real model turned up, so each part reopens the files the previous one established. They are cleanly separated in history (commits 1–8, 9–17, 18–28, 29–30) and can be reviewed in that order.

Root cause is upstream (Magpie/TraceLens/GEAK/IntelliKit/AgentKernelArena), ticket filed:

Not upstream — entirely in-repo. The argv-preflight fix in commit 29 is a misuse of vLLM's FlexibleArgumentParser on our side, not a defect in it: parse_known_args is documented to delegate to argparse, and we were asking the wrong entry point.

[xiaofei-zheng]

Reviewed the full diff. Two blocking issues; everything else I checked held up.

1. --phase-budget-enablement-pct is a new operator flag that nothing enforces

parser.py:1259-1268 adds --max-minutes-enablement-pct / --phase-budget-enablement-pct, whose help text reads "Wall-clock budget cap for ENABLEMENT as a fraction of --max-hours", and cli/__init__.py:1054 maps it into the budget dict. But the if current == PHASE_ENABLEMENT: branch of compute_next_phase (machine_state.py:1996-2012) never calls phase_cap_exceeded — its only call sites are machine_state.py:1750/1801/1919 and phases/kernel.py:962.

Reproduced against this branch:

s.phase = "ENABLEMENT"; s.baseline_tput = 0.0; s.max_minutes = 120
s.enablement = SimpleNamespace(validation_pending=False)

phase_cap_seconds(s, budget_pct={"ENABLEMENT": 0.0001})                       -> 0.72
phase_cap_exceeded(s, budget_pct={"ENABLEMENT": 0.0001}, now_unix=1e12)       -> True
compute_next_phase(s, budget_pct={"ENABLEMENT": 0.0001}, now_unix=1e12, ...)  -> None

The cap is computed, reports exceeded, and no transition follows. An operator who sets the flag to bound bring-up gets no bound and no warning. The PR description's claim that "--phase-budget-enablement-pct still imposes one for an operator who wants it" does not hold.

Either wire phase_cap_exceeded into the ENABLEMENT branch, or drop the flag rather than ship one that parses and does nothing.

2. An explicit --max-hours 2 on resume is silently replaced by the persisted budget

cli/__init__.py:1328:

if float(getattr(args, "max_hours", 0) or 0) == DEFAULT_MAX_HOURS and persisted_minutes > 0:
    args.max_hours = persisted_minutes / 60.0

DEFAULT_MAX_HOURS is 2.0 (parser.py:37), so this equality cannot distinguish "the user did not pass --max-hours" from "the user passed --max-hours 2". Resuming a session whose state.max_minutes is 480 while explicitly asking for 2 hours restores 8 hours and the run goes four times over the requested budget. The function's own docstring says "an explicit flag always wins", and _start_run (coordinator.py:1305-1309) documents the same intent — "a leg that starts with a smaller --max-hours than the session was given must run against the smaller one".

Fix by distinguishing absence from value: give the argument default=None and resolve the default after the restore, or record whether the flag was supplied.

---

Paths checked beyond the two above, all clean:

• every branch of compute_next_phase, including the new ENABLEMENT entry/exit predicate and its interaction with _global_terminal / _closing_phase_terminal;
• EnablementLane round lifecycle: the removed run-deadline guard in _maybe_enqueue_enablement_specialist is backstopped by the deadline.expired() check at the end of each coordinator tick, and _pump_enablement_safely is already skipped while in_closing, so the lane cannot mint rounds past the session clock;
• the deleted close guard / suppression counter: ENABLEMENT_MAX_ATTEMPTS still writes enablement_attempts_exhausted to stop_reason, so a wedged lane retains a terminal path;
• _promote_baseline vs _persist_eval_failure: a sub-floor revalidation is stamped baseline_eval_failed by _stamp_eval_failure_contract, so _is_promotable_result routes it away from the promote path — the rearm branch is not reachable with a live accuracy gate;
• enqueue_targeted_build: attempt_root defaults to "", not None, and build_novelty_key excludes it, so minting the task id early does not break the idempotency key;
• argv_preflight: the probe rebinds parser.error/parser.exit to raise, so every argparse error class (invalid choice, bad type, unrecognized) still emits invalid; the unrecognized arguments: regex only selects which flags the one repair may drop, it does not gate the verdict.

Tests: pytest src/hyperloom/inference_optimizer/tests/test_phase_state_machine.py — 39 passed, 1 collection error from a Windows-only fcntl import, unrelated to this PR.

[ZhengGong-amd]

Both blocking issues are fixed in 01bc91aa7.

1. The budget flag is removed, not wired up. Wiring phase_cap_exceeded into the ENABLEMENT branch would reinstate the cap this branch argues out of existence — a combo that cannot run has nothing to optimise — so --max-minutes-enablement-pct / --phase-budget-enablement-pct and its _build_phase_budget_pct entry are gone. The flag is new on this branch, so nothing depended on it. The PR description sentence you quoted is corrected; that claim was wrong and I had not checked it.

2. --max-hours now carries default=None, and the default is settled after _restore_budget_and_objective has had its chance at the persisted budget, so absence and an explicit 2 are distinguishable rather than merely unequal most of the time:

passed	persisted	result
—	8h	8h
--max-hours 2	8h	2h
--max-hours 8	2h	8h
—	none	2h

_apply_agentx_budget_profile compared against a hand-written 2.0 for the same reason; it reads the constant now. Added test_an_explicit_budget_equal_to_the_default_still_wins.

---

One correction to the clean list. On _promote_baseline: the revalidation_below_floor rearm branch is reachable, so it is not dead code.

The stamping is gated twice on eval-enablement, not just on a live accuracy gate. Both _stamp_eval_failure_contract call sites sit behind _eval_enablement_active (baseline.py:2222 and :2464, helper at :2300), which requires resolve_enablement_mode(state) in {eval, all} (_accuracy_gate.py:134). Under --enablement launch nothing is stamped, _is_promotable_result lets the result through, and a revalidation whose eval returns 0 or no score lands on the rearm branch at writeback.py:3593.

Worth keeping reachable, because the same predicate carries this PR's eval_disabled condition: with --no-eval there is no accuracy to judge, so the revalidation promotes on throughput instead of reopening the window forever.

[xiaofei-zheng]

Re-reviewed at 632912d0 (five new commits since my last pass). Both issues from that pass are fixed correctly — the flag is gone rather than wired up, and --max-hours now carries no argparse default with DEFAULT_MAX_HOURS settled after the restore, covered by test_an_explicit_budget_equal_to_the_default_still_wins.

Blocking: CHANGELOG contradicts the budget table

CHANGELOG.md, in the ENABLEMENT entry:

The other phases keep the percentages they had, so the table now sums to 0.95

They do not. This branch changes PHASE_FRAMEWORK_AGENT 0.40 → 0.38 and PHASE_KERNEL_AGENT 0.50 → 0.47 (machine_state.py:255-256; merge-base 27fc2d9 has 0.40/0.50). The PR description states the change explicitly ("Budget: FRAMEWORK_AGENT 40→38%, KERNEL_AGENT 50→47%"), so the changelog is the copy that is wrong. The sum of 0.95 is right; the sentence explaining how it got there is not.

Non-blocking: default=None inverts the AgentX budget note

Flagging because it came in with the --max-hours fix, not because it blocks — the effect is confined to one advisory line on stderr.

_apply_agentx_budget_profile (cli/__init__.py:985) runs at cli/__init__.py:1657, before the new default settling at 1660-1661, and still tests equality against DEFAULT_MAX_HOURS:

invocation	args.max_hours at line 985	note printed	wanted
no --max-hours (is at the default)	None	no	yes
--max-hours 2 (explicitly chosen)	2.0	yes, "is at its default of 2.0"	no

test_agentx_budget_and_guards.py cannot see it: _budget_args() hardcodes max_hours=2.0 rather than parsing an argv, so the fixture never produces the None the parser now does. Moving the check below line 1661 fixes it.

---

Also checked, clean: the ATOM entry added to STALE_PROCESS_PATTERNS; the retired is_valid_phase_exit_reason assertions; the reworked test_phase_budget_help_quotes_the_real_default; and every remaining reader of args.max_hours (bootstrap.py:298, bootstrap.py:546-558 via _seed_shared_state at line 2145, manifest.py:290, and lines 2184/2335/2443/2460/2472) — all either None-tolerant or reached after the settling on both the fresh-launch and resume paths.

Tests: pytest src/hyperloom/inference_optimizer/tests/test_machine_state.py src/hyperloom/inference_optimizer/tests/test_phase_state_machine.py — 60 passed; the remaining failure and errors are all a Windows-only fcntl import in cli/kb.py, unrelated to this PR.

[xiaofei-zheng]

Pushed 704e0040f to this branch with both items from the review — revert it if you would rather fix them yourself.

• CHANGELOG.md: the entry claimed the other phases keep the percentages they had; it now states PHASE_FRAMEWORK_AGENT 0.40 → 0.38 and PHASE_KERNEL_AGENT 0.50 → 0.47, matching the diff and the PR description.
• cli/__init__.py:985: _apply_agentx_budget_profile runs at 1657, before either path settles the budget, so it sees the parser's None. Testing == DEFAULT_MAX_HOURS inverted the note — silent when nothing was passed, printed when the operator typed 2.0. Now keyed on is None.
• test_agentx_budget_and_guards.py: _budget_args hardcoded max_hours=2.0, which is why the inversion was invisible; it now defaults to the None the parser produces, plus two cases covering both directions of the note. Both fail on the previous condition.

Ran test_agentx_budget_and_guards.py and test_resume_budget_objective.py: 46 passed.

[xiaofei-zheng]

No blocking issues. Both items from the earlier round are resolved — disclosure: by my own commit 704e0040f on this branch, so this approval covers the author's work, not mine; 704e0040f itself is unreviewed by anyone else.

What I checked:

• compute_next_phase ENABLEMENT entry and exit: baseline_tput > 0, validation_pending, enablement_in_flight; the branch deliberately never consults phase_cap_exceeded, consistent with ENABLEMENT having no DEFAULT_PHASE_BUDGET_PCT key. The 0.05 the table no longer allocates falls through to the run's own wall clock, not to a phase that can overrun.
• _pump_enablement_safely is now gated on the phase being ENABLEMENT. The run-deadline guard removed from _maybe_enqueue_enablement_specialist is backstopped by the end-of-tick deadline.expired() check in coordinator.py, so removing it does not let work outlive the deadline.
• Round mutex (rounds.open/held), ENABLEMENT_MAX_ATTEMPTS, and the revalidation window: enablement_attempts_exhausted is the terminal exit and is reachable.
• Accuracy gate: BASELINE_EVAL_FAILED_KEY / _is_promotable_result / _promote_baseline. The sub-floor rearm branch in _promote_baseline is unreachable because a sub-floor result with baseline_eval_failed set is diverted to _handle_unpromotable_result first — dead, not wrong.
• argv preflight: _PARSE_PROGRAM rebinds parser.error/parser.exit to raise, so every argparse error class reaches the invalid path and the drop regex only selects flags that are safe to drop.
• --max-hours sentinel: absence stays None through _restore_budget_and_objective, so a bare --resume-from restores the persisted budget, and an explicit value equal to the default still wins over it. Both paths settle the default afterwards.
• CHANGELOG entry and PR description both match the diff, including the 40→38 / 50→47 budget shift.

Tests run locally (targeted, not the full suite): test_agentx_budget_and_guards.py and test_resume_budget_objective.py, 46 passed. Full CI on this head is green across all six shards.