SCHED-1920: Remove DCGM hpc_job job mapping, join via Slurm exporter

docs/architecture.md

@@ -75,7 +75,7 @@ Soperator consists of the following components:
 - **soperator-activechecks**. This chart is optional. It is used to deploy Kubernetes resources of `kind: ActiveCheck`, which perform active jobs to monitor and validate the state of a Slurm cluster.
 - **soperator-crds**. It deploys the schema of the `SlurmCluster` custom resource, it should always be applied when
   you upgrade the version of Soperator in your cluster.
-- **soperator-dcgm-exporter**. This chart is optional. It is a custom Helm chart with `hpc-jobs-dir` where Slurm stores information about jobs.
+- **soperator-dcgm-exporter**. This chart is optional. It deploys the NVIDIA DCGM exporter with soperator's curated metric set and Slurm-aware relabelings (NVLink instance group and NodeSet labels).
 - **soperator-fluxcd**. This chart is optional. It is an umbrella chart with various components designed for convenient use in a cluster with soperator. It includes `kind: HelmRelease` and `kind: HelmRepository`. This approach was chosen because customizing components through Kustomize is difficult, and Helm dependencies lack flexibility in configuration. Therefore, this umbrella chart was created to deploy HelmReleases with properly configured dependencies.
 - **soperatorchecks**. This chart is optional. It is used to deploy additional controller that runs Slurm and Kubernetes jobs to actively check the state of the Slurm cluster.
 

docs/metrics-pipeline.md

@@ -67,6 +67,7 @@ The exporter applies metric relabeling to drop volatile Kubernetes labels (`pod`
 - Metrics: GPU temperature, power, utilization, memory, errors
 - Scrape Interval: 15s
 - DaemonSet: Runs on nodes with `nvidia.com/gpu.deploy.dcgm-exporter=true`
+- Job attribution: DCGM metrics carry no job labels; per-job GPU views join them with the Slurm exporter's `slurm_node_job` metric on `node_name`, which is added to DCGM series at scrape time from the worker Pod name (see [slurm-exporter.md](slurm-exporter.md))
 
 Connection Example:
 ```bash

docs/slurm-exporter.md

@@ -140,6 +140,20 @@ max by (user_name, nvlink_instance_group, nodeset_name) (
 )
 ```
 
+### Joining DCGM GPU metrics with jobs
+
+DCGM metrics carry no job labels, but they carry `node_name` — the Slurm node name, copied at scrape time from the worker Pod name. Attribute GPU metrics to jobs by joining it with `slurm_node_job`. The join is node-granular — when several jobs share one node, each job inherits the whole node's GPU signal.
+
+```promql
+# Average GPU utilization per running job
+avg by (job_id) (
+  slurm_node_job
+    * on (node_name) group_left()
+  avg by (node_name) (DCGM_FI_DEV_GPU_UTIL)
+)
+and on (job_id) slurm_job_info{job_state="RUNNING"}
+```
+
 ### Controller RPC Metrics
 
 These metrics provide insights into SLURM controller performance, similar to the output of the `sdiag` command, and were implemented to address [issue #1027](https://github.com/nebius/soperator/issues/1027).

helm/slurm-cluster/values.yaml

@@ -874,14 +874,6 @@ slurmScripts:
       enabled: true
       customContent: null
       customConfig: null
-    map_job_dcgm.sh:
-      enabled: true
-      customContent: null
-      customConfig: null
-    unmap_job_dcgm.sh:
-      enabled: true
-      customContent: null
-      customConfig: null
   extra: null
   # extra:
   #   example.sh:

helm/soperator-dcgm-exporter/templates/daemonset.yaml

@@ -24,8 +24,6 @@ spec:
           value: "true"
         - name: DCGM_EXPORTER_COLLECTORS
           value: /etc/dcgm-exporter/dcgm-metrics.csv
-        - name: DCGM_HPC_JOB_MAPPING_DIR
-          value: {{ .Values.dcgmHpcJobMappingDir }}
         - name: NODE_NAME
           valueFrom:
             fieldRef:
@@ -55,30 +53,9 @@ spec:
           name: metrics-config
           readOnly: true
           subPath: dcgm-metrics.csv
-        - mountPath: {{ .Values.dcgmHpcJobMappingDir }}
-          mountPropagation: HostToContainer
-          name: hpc-jobs-dir
       dnsPolicy: ClusterFirst
-      initContainers:
-      - name: create-hpc-jobs-dir
-        command:
-        - sh
-        - -c
-        - |
-          mkdir -p {{ .Values.dcgmHpcJobMappingDir }}
-        env:
-        - name: KUBERNETES_CLUSTER_DOMAIN
-          value: {{ quote .Values.kubernetesClusterDomain }}
-        image: {{ .Values.daemonSet.createHpcJobsDir.image.repository }}:{{ .Values.daemonSet.createHpcJobsDir.image.tag }}
-        imagePullPolicy: {{ .Values.daemonSet.createHpcJobsDir.image.pullPolicy | default "IfNotPresent" }}
-        resources: {}
-        securityContext:
-          privileged: true
-        volumeMounts:
-        - mountPath: /var
-          mountPropagation: HostToContainer
-          name: host-var
       {{- if .Values.validateToolkit }}
+      initContainers:
       - name: toolkit-validation
         args:
         - until [ -f /run/nvidia/validations/toolkit-ready ]; do echo waiting for nvidia container stack to be setup; sleep 5; done
@@ -126,14 +103,6 @@ spec:
           type: ""
         name: run-nvidia
       {{- end }}
-      - hostPath:
-          path: {{ .Values.dcgmHpcJobMappingDir }}
-          type: ""
-        name: hpc-jobs-dir
-      - hostPath:
-          path: /var
-          type: ""
-        name: host-var
       - configMap:
           defaultMode: 420
           items:

helm/soperator-dcgm-exporter/values.yaml

@@ -1,7 +1,5 @@
 kubernetesClusterDomain: cluster.local
 
-dcgmHpcJobMappingDir: /var/run/nebius/slurm
-
 validateToolkit: true
 
 metricsPort: 9400
@@ -16,12 +14,6 @@ daemonSet:
       tag: v24.6.2
       pullPolicy: IfNotPresent
 
-  createHpcJobsDir:
-    image:
-      repository: cr.eu-north1.nebius.cloud/soperator/busybox
-      tag: latest
-      pullPolicy: IfNotPresent
-
   nvidiaDcgmExporter:
     image:
       repository: cr.eu-north1.nebius.cloud/soperator/dcgm-exporter
@@ -43,6 +35,14 @@ serviceMonitor:
       regex: feature_node_kubernetes_io_.*|kubernetes_io_.*|nvidia_com_gpu.*
     - action: labeldrop
       regex: UUID|DCGM_FI_PROCESS_NAME|DCGM_FI_DEV_SERIAL|DCGM_FI_DEV_NAME|job|pci_bus_id|prometheus
+    # The worker Pod name (exported_pod after the scrape resolves the pod-label
+    # conflict) equals the Slurm node name; expose it as node_name so DCGM series
+    # join with the Slurm exporter metrics without label rewriting.
+    - action: replace
+      regex: (.+)
+      sourceLabels:
+        - exported_pod
+      targetLabel: node_name
   relabelings:
     - action: replace
       regex: nvidia-dcgm-exporter

helm/soperator-fluxcd/templates/dcgm-exporter.yaml

@@ -27,7 +27,6 @@ spec:
   upgrade:
     {{- toYaml .Values.observability.dcgmExporter.upgrade | nindent 4 }}
   values:
-    dcgmHpcJobMappingDir: {{ .Values.observability.dcgmExporter.values.hpcJobMapDir }}
     validateToolkit: {{ .Values.observability.dcgmExporter.values.validateToolkit }}
     {{- if .Values.observability.dcgmExporter.values.resources }}
     daemonSet:

helm/soperator-fluxcd/values.yaml

@@ -648,7 +648,6 @@ observability:
         retries: 6
         remediateLastFailure: true
     values:
-      hpcJobMapDir: /var/run/nebius/slurm
       validateToolkit: true
       resources:
         limits:

helm/soperator-monitoring-dashboards/dashboards/cluster_health.json

@@ -3909,7 +3909,7 @@
               },
               "editorMode": "code",
               "exemplar": false,
-              "expr": "with (\n  filters = {}\n)\n# GPU Util\navg(\n  label_move(\n    DCGM_FI_DEV_GPU_UTIL{cluster=~\"$cluster\", filters}\n  , 'hpc_job','job_id')\n) by (job_id)\nand on (job_id)\nslurm_job_info{cluster=~\"$cluster\", \n  filters,\n  job_state=\"RUNNING\",\n  user_name=~\"$user_name\", \n  slurm_partition${partition_match_exp}, \n}",
+              "expr": "with (\n  filters = {}\n)\n# GPU Util\navg(\n  slurm_node_job{cluster=~\"$cluster\"}\n  * on(cluster, node_name) group_left()\n  avg(\n    DCGM_FI_DEV_GPU_UTIL{cluster=~\"$cluster\", filters}\n  ) by (cluster, node_name)\n) by (job_id)\nand on (job_id)\nslurm_job_info{cluster=~\"$cluster\",\n  filters,\n  job_state=\"RUNNING\",\n  user_name=~\"$user_name\",\n  slurm_partition${partition_match_exp},\n}",
               "format": "table",
               "hide": false,
               "instant": true,
@@ -3924,7 +3924,7 @@
               },
               "editorMode": "code",
               "exemplar": false,
-              "expr": "with (\n  filters = {}\n)\n# SM Util\navg(\n  label_move(\n    DCGM_FI_PROF_SM_ACTIVE{cluster=~\"$cluster\", filters}\n  , 'hpc_job','job_id')\n) by (job_id)\nand on (job_id)\nslurm_job_info{cluster=~\"$cluster\", \n  filters,\n  job_state=\"RUNNING\",\n  user_name=~\"$user_name\", \n  slurm_partition${partition_match_exp}, \n}",
+              "expr": "with (\n  filters = {}\n)\n# SM Util\navg(\n  slurm_node_job{cluster=~\"$cluster\"}\n  * on(cluster, node_name) group_left()\n  avg(\n    DCGM_FI_PROF_SM_ACTIVE{cluster=~\"$cluster\", filters}\n  ) by (cluster, node_name)\n) by (job_id)\nand on (job_id)\nslurm_job_info{cluster=~\"$cluster\",\n  filters,\n  job_state=\"RUNNING\",\n  user_name=~\"$user_name\",\n  slurm_partition${partition_match_exp},\n}",
               "format": "table",
               "hide": false,
               "instant": true,
@@ -4734,13 +4734,13 @@
               "disableTextWrap": false,
               "editorMode": "code",
               "exemplar": false,
-              "expr": "floor(\n    avg_over_time(\n  (\n    (\n      (\n        round((count(DCGM_FI_DEV_FB_USED{cluster=~\"$cluster\", hpc_job!=\"\"} > 2) by(hpc_job) / 8) > 2)\n          -\n        round(\n          (count(DCGM_FI_DEV_POWER_USAGE{cluster=~\"$cluster\", hpc_job!=\"\"} > 120) by(hpc_job) / 8)\n            >\n          2\n        )\n      )\n        /\n      round((count(DCGM_FI_DEV_FB_USED{cluster=~\"$cluster\", hpc_job!=\"\", } > 2) by(hpc_job) / 8) > 2)\n    )\n      *\n    100\n  )[1h:1m]\n) >=  70\n)\n  and on(hpc_job)\nlabel_move(slurm_job_info{cluster=~\"$cluster\", job_state=\"RUNNING\"}, 'job_id', 'hpc_job')\nand on (hpc_job)\ncount_over_time( #filter out short durations\navg_over_time((\n  (\n    (\n      (\n        round((count(DCGM_FI_DEV_FB_USED{cluster=~\"$cluster\", hpc_job!=\"\"} > 2) by(hpc_job) / 8) > 2)\n          -\n        round(\n          (count(DCGM_FI_DEV_POWER_USAGE{cluster=~\"$cluster\", hpc_job!=\"\"} > 120) by(hpc_job) / 8)\n            >\n          2\n        )\n      )\n        /\n      round((count(DCGM_FI_DEV_FB_USED{cluster=~\"$cluster\", hpc_job!=\"\"} > 2) by(hpc_job) / 8) > 2)\n    )\n      *\n    100\n  )[1h:1m]\n) > 70\n  and on(hpc_job)\nlabel_move(slurm_job_info{cluster=~\"$cluster\", job_state=\"RUNNING\"}, 'job_id', 'hpc_job')\n)[1h:1m]\n) > 20",
+              "expr": "with (\n  gpus_total = count(DCGM_FI_DEV_FB_USED{cluster=~\"$cluster\"}) by (cluster, node_name),\n  mem_frac = count(DCGM_FI_DEV_FB_USED{cluster=~\"$cluster\"} > 2) by (cluster, node_name) / gpus_total,\n  power_frac = count(DCGM_FI_DEV_POWER_USAGE{cluster=~\"$cluster\"} > 120) by (cluster, node_name) / gpus_total,\n  workers_with_mem = round(sum(slurm_node_job{cluster=~\"$cluster\"} * on(cluster, node_name) group_left() mem_frac) by (job_id) > 2),\n  workers_with_power = round(sum(slurm_node_job{cluster=~\"$cluster\"} * on(cluster, node_name) group_left() power_frac) by (job_id) > 2),\n  underutil_pct = (workers_with_mem - workers_with_power) / workers_with_mem * 100,\n  running_jobs = slurm_job_info{cluster=~\"$cluster\", job_state=\"RUNNING\"}\n)\nfloor(avg_over_time(underutil_pct[1h:1m]) >= 70)\nand on(job_id) running_jobs\nand on (job_id)\ncount_over_time( #filter out short durations\n  (avg_over_time(underutil_pct[1h:1m]) > 70 and on(job_id) running_jobs)[1h:1m]\n) > 20",
               "format": "time_series",
               "fullMetaSearch": false,
               "hide": false,
               "includeNullMetadata": true,
               "intervalFactor": 2,
-              "legendFormat": "{{hpc_job}}",
+              "legendFormat": "{{job_id}}",
               "range": true,
               "refId": "A",
               "useBackend": false,
@@ -5948,7 +5948,7 @@
                 "uid": "${datasource}"
               },
               "editorMode": "code",
-              "expr": "with (\n  filters = {}\n)\navg(\n  label_move(DCGM_FI_DEV_GPU_UTIL{cluster=~\"$cluster\", filters, hpc_job!=\"\"}, 'hpc_job','job_id')\n  * on (job_id) group_left(user_name)\n  slurm_job_info{cluster=~\"$cluster\", \n    filters,\n    job_state=\"RUNNING\",\n    user_name=~\"$user_name\", \n    slurm_partition${partition_match_exp}\n  }\n) by (user_name)",
+              "expr": "with (\n  filters = {}\n)\navg(\n  (\n    slurm_node_job{cluster=~\"$cluster\"}\n    * on(cluster, node_name) group_left()\n    avg(\n      DCGM_FI_DEV_GPU_UTIL{cluster=~\"$cluster\", filters}\n    ) by (cluster, node_name)\n  )\n  * on (job_id) group_left(user_name)\n  slurm_job_info{cluster=~\"$cluster\",\n    filters,\n    job_state=\"RUNNING\",\n    user_name=~\"$user_name\",\n    slurm_partition${partition_match_exp}\n  }\n) by (user_name)",
               "format": "time_series",
               "intervalFactor": 1,
               "legendFormat": "__auto",