Prometheus/PromQL subtract two gauge metrics

Question

I have this gauge metric "metric_awesome" from two different instances. What i want to do, is subtract instance one from instance two like so

metric_awesome{instance="one"} - metric_awesome{instance="two"}

Unfortunately the result set is empty. Has anyone experienced this?

Answer 1

The issue here is that the labels don't match. What you want is:

metric_awesome{instance="one"} - ignoring(instance) metric_awesome{instance="two"}

Answer 2

If anyone searches this wanting to do this for a one-to-many subtraction, have a look at group_right additionally to what was written before.

metric_awesome{instance="one"} - ignoring(instance) group_right metric_awesome{job="compare-instances"}

See also Prometheus dokumentation

Answer 3

By default Prometheus performs a - b in the following way:

1. It selects all the time series for a query.
2. It selects all the time series for b query.
3. It searches pairs of time series at a and b results with identical sets of labels.
4. It calculates the difference between time series in the found pairs with identical sets of labels.

See these docs for more details.

The metric_awesome{instance="one"} - metric_awesome{instance="two"} returns empty result, because all the matching time series on the left side of - contain instance="one" label, while all the time series on the right side of - contain instance="two" label. There are no time series pairs with identical sets of labels here.

This behavior can be changed with on(), ignoring(), group_left() and group_right() modifiers.

For example, metric_awesome{instance="one"} - ignoring(instance) metric_awesom{instance="two"} instructs to ignore instance label during searching for time series pairs with identical labels. This may result in multiple-to-one matching, when multiple time series on one side of - match a single time series on the another side. By default Prometheus returns error in this case. This can be fixed by adding group_left() or group_right() modifiers to - operator:

metric_awesome{instance="one"}
  - ignoring(instance) group_left()
metric_awesome{instance="two"}

or

metric_awesome{instance="one"}
  - ignoring(instance) group_right()
metric_awesome{instance="two"}

See these docs for more details on these modifiers.