Skip to content

nightly regression: modal-torch-latest full suite failed #9095

Closed

Description

@github-actions

The nightly full-suite run at e35282b failed with real test failures.
Failing test files:

  • tests/unit/v1/zero/test_zero3_nvme_offload.py
  • tests/unit/v1/zero/test_zero3_nvme_param_prefetch.py

Run: https://github.com/deepspeedai/DeepSpeed/actions/runs/43846912936

Activity

  1. Achyuthan-S commented on Nov 17, 2026

    @Achyuthan-S
    Contributor

    These six don't point to a commit. They follow which backend ran the suite.

    The failing runs at e35282b all fell back to AWS Spot, and each fails the same six tests:

    The green runs before them all ran in the Modal sandbox, and there these tests were skipped, not passed:

    The AWS image installs libaio-dev, so these are likely the first runs that execute the ZeRO-3 NVMe offload tests at all. Every failing test offloads parameters to NVMe. TestZero3NVMeOffload::test_optimizer[False/True] offloads only the optimizer and goes through the same _skip_if_unsupported() check, and it isn't among the failures.

    The AWS job log keeps only the summary tail, so I couldn't see the tracebacks.

  2. delock commented on Nov 17, 2026

    @delock
    Collaborator

    Hi @xylian86 can you take a look at this failure? Looks like these tests failed on the first them then really run on AWS instance.

  3. xylian86 commented on Nov 17, 2026

    @xylian86
    Collaborator

    @delock These are real failures from #8843, and #9099 fixes them, could you help review #9099?

  4. added a commit that references this issue on Nov 17, 2026
    54c4031
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

nightly-triageAutomated nightly modal-torch-latest triage

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions