Event-to-tag not applying values correctly in prometheus output

[adietrich-ussignal]

When I originally pulled a given metric, the labels generated by gnmic were uniform even though the metric names were distinct. THis caused an issue of 'duplicate time series' within prometheus. I attempted to add unique labels based on metric values using an event-to-tag processor.

processors: 
  add-metric-tag-hwmod: 
    event-to-tag: 
      value-names: 
        - ".*channel-number"
      keep: true
outputs:
  prometheus:
    type: prometheus
    listen: 0.0.0.0:9804
    path: /metrics
    strings-as-labels: True
    expiration: 60s
    timeout: 30s
    debug: False
    event-processors:
      - add-metric-tag-hwmod

subscriptions:
  hwmod_channel_number:
    paths:
      - Cisco-IOS-XR-osa-hwmod-linesys-oper:osa/node-ids/node-id[node-name=0/0/NXR0]/terminal-ampli/flexi-grid-info/channel-number
    mode: stream
    stream-mode: sample
    sample-interval: 60s

When this goes into the Prometheus format, it only applies one of the channel_number values to all entries.

Cisco_IOS_XR_osa_hwmod_linesys_oper_osa_node_ids_node_id_terminal_ampli_flexi_grid_info_0_channel_number{channel_number="13",node_id_node_name="0/0/NXR0",source="<omitted>",subscription_name="hwmod_channel_number"} 12
Cisco_IOS_XR_osa_hwmod_linesys_oper_osa_node_ids_node_id_terminal_ampli_flexi_grid_info_1_channel_number{channel_number="13",node_id_node_name="0/0/NXR0",source="<omitted>",subscription_name="hwmod_channel_number"} 13

The outputs provided in the event format when outputting to a file look as expected:

outputs:
 output1: 
    name: prometheus
    type: file
    filename: subscriptions 
    format: event
    multiline: true
    event-processors: 
      - add-metric-tag-hwmod

    "tags": {
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.0/channel-number": "12",
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.1/channel-number": "13",
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.2/channel-number": "14",
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.3/channel-number": "15",
      "node-id_node-name": "0/0/NXR0",
      "source": "<omitted>",
      "subscription-name": "hwmod_info"
    },
    "values": {
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.0/channel-number": 12,
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.1/channel-number": 13,
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.2/channel-number": 14,
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.3/channel-number": 15
    }

[karimra]

Prometheus output normalizes event tags to their base path when building labels. So all of these tags:

"Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.0/channel-number": "12",
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.1/channel-number": "13",
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.2/channel-number": "14",
      "Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/flexi-grid-info.3/channel-number": "15",

become:

channel-number: "12",
channel-number: "13",
channel-number: "14",
channel-number: "15",

and they overwrite each since labels must be unique.

A similar issue happens with strings-as-labels and was fixed here 808
I can add the fix to tags as well.

Meanwhile you can use a processor like strings to trim the prefix Cisco-IOS-XR-osa-hwmod-linesys-oper:/osa/node-ids/node-id/terminal-ampli/ and change / to _ to change the tags to:

flexi-grid-info.0_channel-number: "12",
flexi-grid-info.1_channel-number: "13",
flexi-grid-info.2_channel-number: "14",
flexi-grid-info.3_channel-number: "15",