Repository navigation
Conversation
There was a problem hiding this comment.
Code Review
This pull request adds error handling to NVML shutdown calls in the GPU collector to prevent crashes. The review suggests changing the warning log in get_nvidia_gpu_info to a debug log to reduce noise and recommends silencing exceptions in the del method to avoid potential AttributeError issues during interpreter shutdown.
Ensure only one process collects hardware metrics by introducing a file-locking mechanism for the monitor. Add _get_monitor_lock_path to compute a stable lock path (scoped to run_id when parallel="shared"), use platform-specific locks (msvcrt on Windows, fcntl elsewhere), persist lock file while monitoring, and release/remove it in cleanup. Add RunStore.parallel to track parallel mode and update SwanLabInitializer to compute and store parallel_mode, enabling metadata/monitor collection for new runs or shared-parallel runs. Update imports and add tests to cover lock path behavior and parallel-mode metadata collection.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements a cross-platform file-locking mechanism to prevent redundant hardware monitoring in multi-process environments, specifically supporting a new parallel="shared" mode. Changes include the addition of msvcrt and fcntl logic for lock management, updates to the SwanLabRun lifecycle for lock acquisition and release, and improved error handling during GPU library shutdown. Review feedback suggests using more idiomatic if statements instead of logical operators for side effects, avoiding the deletion of advisory lock files to prevent potential race conditions, and cleaning up unused exception variables.
|
有多物理节点问题,该方案暂时不适用 |
Description
Demo log exception
Issues: #1532