Synchronize Prometheus data on all pods

Question

I have a question regarding synchronizing Prometheus data. I have 2 pods running (the same Spring Boot service and connected to same Redis Db) in 2 different environments.

When one service updates a metric I would like to enforce that the other one also refreshes it's metric so that they are both displaying the same information. Is there a way to enforce a reload on all services connected? Can this even be done or is it a bad use-case of the metrics?

Thanks!

P.S. The information of that metric represents the latest event that happened on that service. So it makes sense that both services should display the same information.

Answer 1

I think you are trying to solve the problem from the wrong end. Leaving it as it is, PromQL can show you when was the latest event among all instances. At the same time, you keep the possibility to know when the last event was on each instance of the service.

Here is an example for you to get started. Suppose there is a timestamp metric (a Gauge, that holds unixtime of a certain event):

last_event{instance="foo", service="dispatch"} 100
last_event{instance="bar", service="dispatch"} 200
# instead of 100 and 200 there should be the number of seconds since 01.01.1970, but for the sake of simplicity...

With metrics like these you can learn when the last event was with the following query:

max by(service) (last_event)

And the result will be:

{service="dispatch"} 200