'Advantage of avg_over_time(rate()) over rate() by itself?

Usually my queries with counter metrics look like this:

rate(metric_total[5m])
rate(metric_total[$__interval])

But while looking at various companies / teams using Prometheus and Grafana (for example the GitLab infrastructure team) I came across the following construct:

avg_over_time(recording_rule:rate_5m[$__interval])

So I would like to know: Is there an advantage to the second approach?

Here is a concrete example: https://dashboards.gitlab.com/d/frontend-main/frontend-overview?orgId=1&viewPanel=23

prometheusgrafanapromql


Solution 1:[1]

The 2nd approach doesn't require you to have recording rules for every possible interval over which you'd like an average rate, saving resources.

Solution 2:[2]

The avg_over_time(rate(metric_total[5m])[$__interval:]) calculates average of average rates. This isn't a good metric, since average of averages doesn't equal to average. So the better approach would be to calculate rate(metric_total[$__interval]) - it returns the real average per-second increase rate for metric_total over the given lookbehind window $__interval.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 brian-brazil
Solution 2 valyala

Related Questions