'How to execute a query with two metrics in Prometheus?

I am using Prometheus to query metrics from Apache Flink. I want to measure the number of records In and Out per second of a Map function. When I query two different metrics in Prometheus, the chart only shows one of them.

flink_taskmanager_job_task_operator_numRecordsInPerSecond{operator_name="Map"} 
or flink_taskmanager_job_task_operator_numRecordsOutPerSecond{operator_name="Map"}

Does not matter if I change the operator or to and. The chart shows only the first (flink_taskmanager_job_task_operator_numRecordsInPerSecond). I also have tried to edit the Prometheus config file /etc/prometheus/prometheus.yml but I don't have too much experience on Prometheus and there is something wrong in my configuration. I was basing my solution on this post.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100']   
  - job_name: 'flink'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9250', 'localhost:9251', '192.168.56.20:9250']
    metrics_path: /
# HOW TO ADD THE OPERATOR NAME ON THE METRIC NAME?
    metric_relabel_configs:
      - source_labels: [__name__]
      regex: '(flink_taskmanager_job_task_operator)_(\w+)'
      replacement: '${2}'
      target_label: pool
      - source_labels: [__name__]
      regex: '(flink_taskmanager_job_task_operator)_(\w+)'
      replacement: '${1}_bytes'
      target_label: __name__

prometheus

Solution 1:^[1]

It is possible to select multiple metric names with a single PromQL query by using a regular expression filter on __name__ label:

{__name__=~"flink_taskmanager_job_task_operator_numRecords(In|Out)PerSecond",operator_name="Map"}

See docs about the __name__ label here.

There is another solution when using Prometheus-compatible query engine such as MetricsQL by using union function:

union(
flink_taskmanager_job_task_operator_numRecordsInPerSecond{operator_name="Map"},
flink_taskmanager_job_task_operator_numRecordsOutPerSecond{operator_name="Map"}
)

Solution 2:^[2]

First of all, for more complex graphing you should definitely investigate Grafana. The built-in Prometheus graphs are useful eg. for debugging, but definitely more limited. In particular one graph will only display the results of one query.

Now for a hack that I definitely do not recommend:

flink_taskmanager_job_task_operator_numRecordsInPerSecond{operator_name="Map"}
or
label_replace(flink_taskmanager_job_task_operator_numRecordsOutPerSecond{operator_name="Map"}, "distinct", "foo", "job", ".*")

Since, as documented

vector1 or vector2 results in a vector that contains all original elements (label sets + values) of vector1 and additionally all elements of vector2 which do not have matching label sets in vector1.

you can add a new label that is not present in the labels on the first vector to the second vector and thus keep all elements from both.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	valyala
Solution 2	Micha? Politowski

'How to execute a query with two metrics in Prometheus?

Solution 1:[1]

Solution 2:[2]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]