Skip to content

fix: pynvml shutdown error - #1534

Closed
Nexisato wants to merge 6 commits into
legacyfrom
fix/pynvml-release-error
Closed

Nexisato wants to merge 6 commits into
legacyfrom
fix/pynvml-release-error

Conversation

@Nexisato

@Nexisato Nexisato commented Apr 3, 2026 •

Copy link
Copy Markdown
Collaborator

Description

  • fix: pynvml shutdown exception

Demo log exception

 Exception ignored in: <function GpuCollector.__del__ at 0x7f34a2572d40>
Traceback (most recent call last):
  File "/usr/local/miniconda3/envs/py312/lib/python3.12/site-packages/swanlab/data/run/metadata/hardware/gpu/nvidia.py", line 267, in __del__
    pynvml.nvmlShutdown()
  File "/usr/local/miniconda3/envs/py312/lib/python3.12/site-packages/pynvml.py", line 2970, in nvmlShutdown
    _nvmlCheckReturn(ret)
  File "/usr/local/miniconda3/envs/py312/lib/python3.12/site-packages/pynvml.py", line 1083, in _nvmlCheckReturn
    raise NVMLError(ret) 

Issues: #1532

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds error handling to NVML shutdown calls in the GPU collector to prevent crashes. The review suggests changing the warning log in get_nvidia_gpu_info to a debug log to reduce noise and recommends silencing exceptions in the del method to avoid potential AttributeError issues during interpreter shutdown.

Comment thread swanlab/data/run/metadata/hardware/gpu/nvidia.py Outdated
Comment thread swanlab/data/run/metadata/hardware/gpu/nvidia.py Outdated
Nexisato added 2 commits April 3, 2026 17:40
Ensure only one process collects hardware metrics by introducing a file-locking mechanism for the monitor. Add _get_monitor_lock_path to compute a stable lock path (scoped to run_id when parallel="shared"), use platform-specific locks (msvcrt on Windows, fcntl elsewhere), persist lock file while monitoring, and release/remove it in cleanup. Add RunStore.parallel to track parallel mode and update SwanLabInitializer to compute and store parallel_mode, enabling metadata/monitor collection for new runs or shared-parallel runs. Update imports and add tests to cover lock path behavior and parallel-mode metadata collection.
@Nexisato

Nexisato commented Apr 3, 2026

Copy link
Copy Markdown
Collaborator Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a cross-platform file-locking mechanism to prevent redundant hardware monitoring in multi-process environments, specifically supporting a new parallel="shared" mode. Changes include the addition of msvcrt and fcntl logic for lock management, updates to the SwanLabRun lifecycle for lock acquisition and release, and improved error handling during GPU library shutdown. Review feedback suggests using more idiomatic if statements instead of logical operators for side effects, avoiding the deletion of advisory lock files to prevent potential race conditions, and cleaning up unused exception variables.

Comment thread swanlab/data/run/main.py
Comment thread swanlab/data/run/main.py Outdated
Comment thread swanlab/data/run/metadata/hardware/gpu/nvidia.py Outdated
@Nexisato Nexisato closed this Apr 8, 2026
@Nexisato

Nexisato commented Apr 8, 2026 •

Copy link
Copy Markdown
Collaborator Author

有多物理节点问题,该方案暂时不适用

@Nexisato
Nexisato deleted the fix/pynvml-release-error branch April 24, 2026 04:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant