'Prometheus collect metrics from different microservices (metrics_paths) but same target (ip and port)

I want to collect metrics from different microservices from application my_app.

I want to be able to group them, e.g. get sum of requests in all microservices on one environment or compare metrics from same microservice on different environments.

dev_env.com/api/a/actuator/prometheus
dev_env.com/api/b/actuator/prometheus
dev_env.com/api/c/actuator/prometheus
test1_env.com/api/a/actuator/prometheus
test1_env.com/api/b/actuator/prometheus
test1_env.com/api/c/actuator/prometheus

Currently I plan to configure prometheus with as many jobs as microservices like this:

- job_name: "dev_a"
  metrics_path: "/api/a/actuator/prometheus"
  static_configs:
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "a"

- job_name: "dev_b"
  metrics_path: "/api/b/actuator/prometheus"
  static_configs:
    - targets: ["dev_env.com"]
      labels:
        app: "my_app"
        env: "dev"
        microservice: "b"
...

Is there a way to solve this problem with only one job? Is it correct approach to have one or multiple jobs in such a scenario? In reality there are really a lot of microservices and few environments.

Goal: have metrics with fields env, microservice, app. Then i will be able to easily manipulate with them. If there would be solution with one job, then i don't need app field, because job field would replace it. Now i need app to distinguish metrics from my_app and other metrics, e.g. kafka, servers.

I saw similar problem on stackoverflow but the difference here is I have the same ip and port for all microservices. Also I didn't find easy way for specifying multiple metrics_paths for one job.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source