You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
nightly regression: modal-torch-latest full suite failed #9095
The AWS image installs libaio-dev, so these are likely the first runs that execute the ZeRO-3 NVMe offload tests at all. Every failing test offloads parameters to NVMe. TestZero3NVMeOffload::test_optimizer[False/True] offloads only the optimizer and goes through the same _skip_if_unsupported() check, and it isn't among the failures.
The AWS job log keeps only the summary tail, so I couldn't see the tracebacks.
Activity
Achyuthan-S commentedon Nov 17, 2026
These six don't point to a commit. They follow which backend ran the suite.
The failing runs at e35282b all fell back to AWS Spot, and each fails the same six tests:
op_builder/dc.py. That run also fails one AutoTP test.The green runs before them all ran in the Modal sandbox, and there these tests were skipped, not passed:
The AWS image installs
libaio-dev, so these are likely the first runs that execute the ZeRO-3 NVMe offload tests at all. Every failing test offloads parameters to NVMe.TestZero3NVMeOffload::test_optimizer[False/True]offloads only the optimizer and goes through the same_skip_if_unsupported()check, and it isn't among the failures.The AWS job log keeps only the summary tail, so I couldn't see the tracebacks.
delock commentedon Nov 17, 2026
Hi @xylian86 can you take a look at this failure? Looks like these tests failed on the first them then really run on AWS instance.
xylian86 commentedon Nov 17, 2026
@delock These are real failures from #8843, and #9099 fixes them, could you help review #9099?
ZeRO-3 NVMe offload: give swapped-out parameter partitions the model …