SCHED-1920: Remove DCGM hpc_job job mapping, join via Slurm exporter - #2923
Open
theyoprst wants to merge 2 commits into
Open
SCHED-1920: Remove DCGM hpc_job job mapping, join via Slurm exporter#2923theyoprst wants to merge 2 commits into
theyoprst wants to merge 2 commits into
Conversation
theyoprst
marked this pull request as draft
September 1, 2026 18:04
theyoprst
force-pushed
the
SCHED-1920/0
branch
from
September 2, 2026 10:42
fbe4445 to
cc6ccbd
Compare
theyoprst
marked this pull request as ready for review
September 2, 2026 14:45
ali-sattari
reviewed
Sep 2, 2026
| slurm_node_job | ||
| * on (node_name) group_left() | ||
| avg by (node_name) ( | ||
| label_replace(DCGM_FI_DEV_GPU_UTIL, "node_name", "$1", "exported_pod", "(.*)") |
Collaborator
There was a problem hiding this comment.
Not directly related to objective of this PR, but since there are a lot of changes here, why not add node_name label to DCGM_* metrics to simplify such joins? we can do it next to other relabeling rules.
| }, | ||
| "editorMode": "code", | ||
| "expr": "avg(DCGM_FI_DEV_SM_CLOCK{cluster=~\"$cluster\", exported_pod=~\"$worker\",hpc_job=~\"$slurm_job\"}*1000000)", | ||
| "expr": "avg(DCGM_FI_DEV_SM_CLOCK{cluster=~\"$cluster\", exported_pod=~\"$worker\"}*1000000)", |
Collaborator
There was a problem hiding this comment.
Removing job filter makes correlating specific job's lifetime more difficult in these panels. It is not a blocker, but something we lose. Do we have other panels or dashboards to see DCGM metrics per job (i.e. only specifically for duration of that job)?
The label multiplied DCGM series count by the number of jobs, and DCGM metrics dominate storage on GPU clusters. Remove the DCGM-Slurm job-mapping integration: the map/unmap_job_dcgm prolog/epilog scripts, the DCGM_HPC_JOB_MAPPING_DIR env, the hpc-jobs-dir/host-var hostPath mounts, the create-hpc-jobs-dir init container, and the hpcJobMapDir fluxcd value. Instead, expose node_name on DCGM series (metric relabeling copies exported_pod, the worker pod name, which equals the Slurm node name) and migrate the dashboards to join with the Slurm exporter's slurm_node_job/slurm_job_info; GPU allocation panels use slurm_node_info state_base as the allocation test. Job attribution becomes node-granular.
theyoprst
force-pushed
the
SCHED-1920/0
branch
from
September 2, 2026 15:21
a0a25de to
5a06109
Compare
Filtering by the $worker variable alone shows the node's GPU metrics for the whole time range, losing what the hpc_job label used to provide: series present only while the job ran. Restore it by intersecting each per-job DCGM expression with slurm_node_job{job_id=~"$slurm_job"}, which exists exactly for the job's runtime on each node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DCGM metrics dominate storage on GPU clusters, and the
hpc_joblabel multiplies DCGM series count by the number of jobs. The label and its plumbing (prolog/epilog scripts, hostPath mounts) exist only to attribute GPU metrics to jobs in dashboards.Solution
map_job_dcgm.sh/unmap_job_dcgm.shprolog/epilog scripts fromslurm-clusterDCGM_HPC_JOB_MAPPING_DIR, thehpc-jobs-dir/host-varhostPath mounts and thecreate-hpc-jobs-dirinit container fromsoperator-dcgm-exporter, andhpcJobMapDirfromsoperator-fluxcdslurm_node_job/slurm_job_infoonexported_pod<->node_name; GPU allocation panels useslurm_node_infostate_baseinstead of thehpc_jobpresence testdocs/slurm-exporter.mdJob attribution becomes node-granular: when several jobs share one node, each job inherits the whole node's GPU signal.
Testing
Validated dashboards on a dev cluster
Release Notes
Breaking: DCGM metrics no longer carry the
hpc_joblabel. Per-job GPU dashboards and queries should join with the Slurm exporter'sslurm_node_jobmetric instead (node-granular attribution).