Skip to content

GEAK fallback can promote candidates rejected by native accuracy validation #1433

Open

Description

@iraj465

Hyperloom's GEAK fallback can promote a throughput result after native validation rejects accuracy. This reproduces on the former PR #1428 head ecf07f0b7bd8ffb2ae65bcafea1927e0275d1d8f using the actual Coordinator, native sweep and promotion methods with CPU fixture subprocesses.

With baseline accuracy 0.94 and current-best throughput 110, missing evaluation or fresh accuracy 0.2 can still lead to throughput 150 being promoted and a 50% validated gain recorded. Native quality controls reject both. A healthy-quality control and a failed-process control establish the intended positive and negative paths.

The route has two defects:

  • Explore can emit per_variant_outcomes with REVERT / accuracy_drop or REVERT / accuracy_unavailable, no winner, and no output throughput while the task itself succeeds because it produced measurements. GEAK writeback treats missing throughput as inconclusive and invokes another harness.
  • The fallback sweep drops fresh evaluation artifacts, and _validate_geak_via_geak_harness checks throughput and GEAK's claimed speedup without applying the native accuracy requirement.

The correction is published in existing PR #1428 at f89eedaeff3a4499d7f068394c05aff633bba841 and awaits review/merge. It:

  1. Preserves the matching current-round native accuracy REVERT, closes the candidate with the existing no-promote tombstone, and withdraws provisional journey KEEPs.
  2. Reads evaluation only from the fresh replay workspace through the existing native parse_eval_results function.
  3. Requires valid fresh accuracy and the unchanged native drop tolerance when a positive baseline accuracy is recorded. Old result or throughput-summary scalar accuracy cannot substitute for evaluation artifacts.
  4. Carries the accepted fresh accuracy into the optimization stack, which CLOSE already reads.

The same sweep has an independent return-configuration defect: it ignores authoritative accepted_config.env_map. The correction reuses the existing shared normalizer, with the actual shell transport correction published in AMD-AGI/GEAK#452 at 812819d82f6baf6ff9fc694eeaff33aba06a1ca5. Twelve real Hyperloom → copied GEAK adapters → Python-S decoder → child-environment controls pass. Parser-only round trips are insufficient.

Final immutable candidate validation: 20,719 full-suite tests passed, 43 skipped, eight expected failures and 15 passing subtests, plus whole-repository Ruff, formatting, REUSE and whitespace checks. Independent source review and 128 focused tests pass. The actual-source reproduction retains 13 corrected cases, fresh accuracy-retention controls, empty-map no-material closure and artifact-only fallback controls. Original vulnerable-source controls and all outputs are retained. No accuracy or performance threshold changed. Final receipt SHA256 688bc268081e0acd1c22725ef5d11fd99e8b9208e3e0f96d0940062226df1bec.

This establishes a current control-flow defect. It does not establish that this exact route caused the historical MiniMax KEEP with degraded accuracy, whose exact executable/state remain incomplete. It is also not a GPU performance or dashboard-recovery claim.

Evidence under /home/sapmajum/work/geak_ci_delivery_20260907/composed_validation/hyperloom_fallback_accuracy_audit:

  • Original replay receipt SHA256 beed72623418c17dd16a4e5bf1f94ab26f744348d73fdede0d5083b962b1ca93.
  • Corrected-source 13-case receipt SHA256 50412445696274f15429782d6c38f0d7c8a305ccb189036ba46eab1577d53173.
  • Independent review SHA256 8d6579c855d937a436e99b1c58ff631d75d21d2449249a7ff6029ded0c74f778.
  • Fresh-accuracy stack-retention receipt SHA256 1c10c10972f2e948c5dee9c737c902688222989de810fb6b68526f2508659aad.

Activity

  1. chaojhou commented on Oct 20, 2026

    @chaojhou
    Collaborator

    Checked against current main: both defects are fixed by #1428 (bacd8e7) and still hold after the later writeback refactors.

    1. WritebackCollaborator.promote_to_shared_state (loop/writeback.py) reads per_variant_outcomes. A matching current-round REVERT with accuracy_drop, accuracy_unavailable or intvty_regression closes the candidate as no-promote and never reaches the GEAK-harness fallback.
    2. validate_geak_via_geak_harness requires a valid fresh replay accuracy whenever a positive baseline accuracy is recorded and applies the native accuracy_passed tolerance. Missing or failing accuracy returns validated: False, status: "no_promote".

    Pinned by test_geak_gain_alignment.py::test_2b_native_revert_is_conclusive, ::test_geak_harness_rejects_missing_or_failed_fresh_accuracy and ::test_geak_harness_accepts_fresh_accuracy_within_native_tolerance, and by test_geak_sweep_units.py::test_sweep_replays_normalized_return_environment_and_fresh_accuracy.

    Nothing is left to do here, so this issue will close on its own. If a run on current main shows the problem again, please open a new issue with the session and head.

  2. github-actions commented on Oct 22, 2026

    @github-actions

    This issue has been quiet for about 2 days since our last reply.

    It will be closed automatically in 1 day if there is no further activity.
    If you are still working on this, please leave a comment with any update — that will reset the timer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

awaiting-author-responseWaiting for issue author to reply (auto-managed)staleNo activity, scheduled for auto-close

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions