Skip to content

Fix Laguna Evalution runtime configuration - #2961

Merged
Qubitium merged 1 commit into
mainfrom
agent/laguna-eval-runtime-fix
Jul 22, 2026
Merged

Fix Laguna Evalution runtime configuration#2961
Qubitium merged 1 commit into
mainfrom
agent/laguna-eval-runtime-fix

Conversation

@Qubitium

Copy link
Copy Markdown
Collaborator

Summary

  • treat legacy random_seed as an Evalution engine option and normalize it to Evalution's canonical seed
  • stop the model-test loader from passing both seed and random_seed, preventing random_seed from leaking into LagunaForCausalLM.__init__
  • run Laguna S 2.1 GSM8K Platinum through the chat-template path so the checkpoint's required thinking default is active

Root cause

The Evalution adapter recognized seed but not random_seed, so the latter remained in Model.model_kwargs and was forwarded through GPTQModel.load() to the Hugging Face model constructor. Separately, Laguna S 2.1's evaluation config used a plain prompt even though its reasoning mode is selected by the chat template and declared as default_chat_template_kwargs={"enable_thinking": true} in generation_config.json.

This change keeps seed canonical, accepts random_seed only as a backwards-compatible engine alias, and enables chat-template evaluation specifically for Laguna S 2.1.

Validation

  • CUDA_VISIBLE_DEVICES=2,3 PYTHON_GIL=0 pytest -q tests/test_eval_loader_args.py tests/test_model_test_helpers.py tests/test_laguna_support.py — 18 passed
  • ruff check on all changed files — passed
  • git diff --check — passed
  • live Laguna S 2.1 GPTQModel checkpoint load/request-preparation smoke on physical GPUs 2 and 3 with CPU spill — engine seed 38, empty model kwargs, EOS [2, 24], </assistant> token [24], and prompt suffix <assistant><think>

Companion Evalution fix: ModelCloud/Evalution#128

@Qubitium
Qubitium marked this pull request as ready for review July 22, 2026 15:28
@Qubitium
Qubitium merged commit 7810b6c into main Jul 22, 2026
6 checks passed
@Qubitium
Qubitium deleted the agent/laguna-eval-runtime-fix branch July 22, 2026 15:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant