'Expression in Prometheus to determine constant growth over time

We get metrics about the number of messages in queues from our instance of ActiveMQ Artemis 2.10.0 to Prometheus, and I need to be notified when for a certain amount of time (let's say 8 hours) the queue grows and does not decrease (usually this indicates a problem with the service that pulls messages from queues).

Like this:

enter image description here

But if I see something like this in the image below i.e. peak growth followed by a decrease, then the alert should not be triggered:

enter image description here

Now I use this expression, but sometimes it does not work correctly due to large growth spurts even with a subsequent decrease:

floor((predict_linear(artemis_message_count{job="activemq",queue=~".*"}[24h], 3600 * 24 * 1))) - max_over_time(artemis_message_count{job="activemq",queue=~".*"}[24h]) > 0

Can't figure out which expression is better to use in order to have fewer fake alerts. Would be grateful for a hint.

prometheusgrafanaactivemq-artemispromqlprometheus-alertmanager


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions