prometheus check if a metric is present

Question

I have alerts based on a metric which in my case, sometimes, may disappear. Let's say the metric as up(env=prod) and up(env=staging). I have an alert based on the value of this metric. Now, I want to trigger another alert if up(env='staging') is not present. I can sum the metric by env and look at the value but it does not tell me which env is missing.

Answer 1

You can do this with absent(up{env="staging"})

Answer 2

The absent function returns non-empty result if the inner time series selector doesn't match any time series. If the series selector matches at least a single time series, then absent() returns nothing. For example, if you have the following metrics:

up{env="staging"}
up{env="prod"}

Then absent(up) returns nothing if up{env="staging"} stops receiving new samples, while up{env="prod"} continues receiving new samples. This means you need to define a separate absent() query per each monitored time series:

absent(up{env="staging"})
absent(up{env="prod"})

This doesn't scale well when the number of monitored time series increases. There is no good generic solution for this issue in Prometheus, but it can be solved with the following single MetricsQL query in VictoriaMetrics:

lag(up[24h]) > 5m

This query returns all the up time series, which received at least a single sample during the last 24 hours, but didn't receive new samples during the last 5 minutes. See docs for lag() function.