Skip to content

Sharded fine-tune path cannot emit Hugging Face eval_loss #778

Open

Description

@DanielAsadi

eval_strategy is accepted on TrainingConfig, but the sharded dataset path cannot actually produce eval_loss.

  1. DatasetFactory.build() asserts eval_strategy == "no" and always returns eval_dataset=None.
  2. ShardedMixtureDataset iterates forever, so HF eval would never finish.
  3. Accelerate prepares the eval iterable dataloader and broadcasts CPU objects through NCCL.
  4. Even if eval runs, Trainer.prediction_step drops the loss: Gr00tN1d7 has empty label_names and no return_loss argument, so logs show eval_runtime only.

Training with eval_strategy: "no" is unaffected.

Activity

  1. DanielAsadi commented on Sep 22, 2026

    @DanielAsadi
    Author

    I opened a PR for this: #777

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions