prometheus can not send firing alert to alertmanager

Question

I use prometheus as a event monitor, which means when something happens in my system, for example: the system upgrade successfully or some error happens. The system has 2 different events, one is called "server upgrade successfully", and another one is called "server upgrade report". Everytime when the server upgrade successfully, alertmanager can get "server upgrade report" alert from prometheus, but can not get "server upgrade successfully" alert. Here is the prometheus alert rules, the first one is "server upgrade successfully" alert rule, second one is "server upgrade report" alert rule.

 groups:
  - name: server_upgrade_succeed_rule_group
    rules:
      - alert: server_upgrade_succeed_rule_alert
        expr: SERVICE_UPGRADE_SUCCESS == 1
        for: 0s
        annotations:
          level: "info"
          category: "app_event"
          rule_id: "server_upgrade_succeed_rule"
          component_id: ""
          summary: "server upgrade successfully"
          description: "{{ $labels.controller_name }} {{ $labels.host_ip }}{{ $labels.service_name }}{{ $labels.end_at }}{{ $labels.original_version }}{{ $labels.target_version }}"


groups:
  - name: server_upgrade_report_rule_group
    rules:
      - alert: server_upgrade_report_rule_alert
        expr: SERVICE_UPGRADE_REPORT == 1
        for: 0s
        annotations:
          level: "info"
          category: "app_event"
          rule_id: "server_upgrade_report_rule"
          component_id: ""
          summary: "server upgrade report"
          description: "{{ $labels.controller_name }}{{ $labels.service_name }}{{ $labels.service_name }}{{ $labels.total_count }}{{ $labels.succeed_count }}{{ $labels.failed_count }}"

the prometheus config is like this

global:
  scrape_interval: 5s
  evaluation_interval: 5s

what confuses me most is that i think the 2 metrics are totally same, it doesn't make sense that one can be send to alertmanager but another can not.

Hope some brilliant guy can help me.

Answer 1

you can install alertmanager and into config u can set the integration to send requests or notifications to send an alert for example, I used webhook integration and write easy code with express node js to receive the request and notify someone (https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver)

in the alertmanager config, I set this config

receivers:
  - name: 'web.hook'
    webhook_configs:
      - url: 'http://127.0.0.1:8090/webhook'
        send_resolved: true

when some alert trigger request POST method to my URL

I hope to solve your question :)