Repository navigation
Feat/system - #1527
Merged
Merged
Feat/system#1527
Conversation
Add a collection of metadata utilities and refactor internal system types to improve metadata collection and monitoring integration. - Add new metadata modules: runtime, git, conda, requirements, hardware (stub) under swanlab/utils/metadata with functions to collect OS, Python, cwd, hostname, command, git info, conda export and pip/alternative package listings. - Export new metadata helpers from swanlab/utils and expose them via swanlab/utils/metadata/__init__.py. - Enrich MetadataSettings docstrings in sdk/internal/settings/metadata.py to clarify semantics and interactions (notably hardware vs monitor behavior) and describe runtime/requirements/conda/git options. - Rename SystemEnvironment -> SystemShim and replace platform field with a more specific slug (linux, macos-intel, macos-arm, windows); update SystemInfo to use shim and adapt from_snapshot logic to detect macOS Apple Silicon. - Update helper.catch_and_return_none to accept an optional on_error callback and invoke it when exceptions are caught. - Remove unused swanlab/utils/metadata/typings.py. These changes centralize metadata collection, improve clarity on configuration semantics, and make the system-monitoring shim more explicit about platform variants.
Move metadata collectors from swanlab/utils/metadata into swanlab/sdk/internal/run/system/environment and adapt APIs and types accordingly. Rename several get_metadata_* helpers to simpler get*_ or get(), add a new HardwareSnapshot type and replace flat MetadataSnapshot fields with a hardware subfield. Rename SystemInfo -> SystemEnvironment (with conda/requirements fields) and extend SystemShim to handle an "unknown" slug and hardware-driven cpu/memory/accelerator flags. Adjust settings: MetadataSettings -> EnvironmentSettings and settings.metadata -> settings.env. Remove old utils.metadata and utils.monitor init modules and trim swanlab.utils exports. Add a skeleton environment/__init__.py and delete the old system README. Overall this refactors where system/environment metadata lives and updates types and settings to match the new layout.
Refactor environment snapshotting: generate() now conditionally collects git, runtime, conda and requirements snapshots via settings.environment flags and delegates hardware to a new hardware_vendor package. Add safe collectors for conda, requirements and rich runtime info with catch_and_return_none wrappers and console error logging; tighten git error logging to error level. Remove legacy hardware.py and introduce hardware_vendor/* placeholders for many vendors (nvidia, amd, cambricon, huawei, iluvatar, kunlunxin, metax, moorethreads, apple, cpu, memory). Rename Settings.env -> Settings.environment and update AcceleratorVendor literals (rename "nvml" to "nvidia" and add other vendor identifiers).
Introduce a new hardware_vendor subsystem and refactor core typings and environment logging. - Add hardware vendor module: README, apple.py and cpu.py implementations for Apple Silicon and generic CPU detection, plus unit tests for Apple Silicon detection. - Add typings for hardware vendors (swanlab/sdk/typings/run/system/hardware_vendor.py) and adapt system typings: rename AppleChipSnapshot -> AppleSiliconSnapshot and update references in system models. - Move CoreProtocol/CoreEnum into swanlab/sdk/typings/core.py (rename from internal/core.py) and update imports to use the new typings module. - Reintroduce create_core factory in swanlab.sdk.internal.run.__init__ to build CorePython by default. - Replace noisy console.error calls with console.debug in several environment info collectors (conda, git, requirements, runtime) to reduce log noise on failures. These changes modularize hardware vendor detection, clarify typing locations, and reduce log verbosity for environment probes.
Implement a generic Memory hardware vendor: add swanlab/sdk/internal/run/system/hardware_vendor/memory.py with _bytes_to_snapshot and Memory class that collects total RAM across Linux (/proc/meminfo), macOS (sysctl hw.memsize) and Windows (wmic), with safe error handling via catch_and_return_none and debug logging. Update pyproject.toml target-version to py39. Add comprehensive unit tests for CPU and Memory collectors under tests/unit/... to validate platform-specific parsing, edge cases, and fallback behaviors.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request upgrades the project's target version to Python 3.9 and introduces a robust system information collection framework. It implements structured data models and collectors for hardware (CPU, Memory, Apple Silicon, and various accelerators), runtime environments, Git metadata, and Python dependencies (supporting pip, conda, uv, and pixi). Key architectural changes include refactoring the core factory logic and enhancing the catch_and_return_none utility with error callbacks. Review feedback highlights several improvement opportunities: fixing Git SSH URL parsing logic, ensuring get_commit works in detached HEAD states, adding timeouts to subprocess calls in the Apple Silicon collector, and making memory unit parsing more resilient to different unit strings.
Comment on lines
+55
to
+65
| branch = get_branch() | ||
| if not branch: | ||
| return None | ||
| result = subprocess.run( | ||
| ["git", "rev-parse", branch], | ||
| capture_output=True, | ||
| text=True, | ||
| timeout=5, | ||
| check=True, | ||
| ) | ||
| return result.stdout.strip() |
Contributor
There was a problem hiding this comment.
get_commit 函数目前依赖于 get_branch 的结果。如果仓库处于“分离头指针”（detached HEAD）状态，get_branch 将返回空字符串，导致 get_commit 返回 None。建议直接使用 git rev-parse HEAD 来获取当前提交的哈希值，这样在任何状态下都能正确获取。
result = subprocess.run(
["git", "rev-parse", "HEAD"],
capture_output=True,
text=True,
timeout=5,
check=True,
)
return result.stdout.strip()Add comprehensive unit tests for environment collectors (conda, git, requirements, runtime). Normalize SSH Git URLs in parse_git_url by converting git@... to https using PurePosixPath and a single ':'->'/' replacement; leave non-SSH URLs unchanged. Adjust runtime error handling: remove the broad decorator from get_command and apply a more specific catch decorator to _get_command_linux (with a clearer debug message). Also add the necessary import for PurePosixPath.
Make get_commit() resolve the latest commit by using `git rev-parse HEAD` (remove dependency on branch lookup) and add `timeout=5` to subprocess calls. Also add a timeout to the macOS `system_profiler` invocation to avoid potential hangs; no behavior changes beyond these robustness fixes and minor formatting.
Use unittest.mock.patch with create=True when patching platform.freedesktop_os_release in tests to avoid AttributeError on Python builds where the attribute isn't present. Updates two test cases in tests/unit/sdk/internal/run/system/environment/test_environment_runtime.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
重构了系统信息采集部分，目前仅完成了对系统信息的模型约定和CPU、Memory、Apple的采集