'define variable in alertmanager configuration yaml file to forward prometheus alerts to slack
I am stuck with a basic YAML problem but I am not sure what's the proper way around it and YAML isn't my strong suit.
I am trying to get the below code to work. The "slack_configs:" section is where I get an error. Whatever I put under the "templates.tmpl" section does not seem to be picked up.
Here's the code:
alertmanager:
baseURL: https://alertmanager.acme.com/
ingress:
enabled: "true"
ingressClassName: nginx
annotations:
kubernetes.io/tls-acme: "true"
hosts:
- alertmanager.acme.com
tls:
- secretName: alertmanager.acme.com
hosts:
- alertmanager.acme.com
## alertmanager ConfigMap entries
##
alertmanagerFiles:
alertmanager.yml:
global:
slack_api_url: https://hooks.slack.com/services/ACME
opsgenie_api_key: 1234
route:
group_wait: 10s
group_interval: 1m
repeat_interval: 2h
receiver: slack
routes:
- receiver: slack
repeat_interval: 1m
continue: true
continue: true
- receiver: OpsGenie
repeat_interval: 1m
receivers:
- name: slack
slack_configs:
- channel: '#alertmanager'
send_resolved: true
title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}{{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .alertname }}{{ end }}{{ end }}'
text: |-
*Cluster:* {{ template "clusterName" . }}
{{ range .Alerts }}• (<{{ with .Labels }}{{template "alertmanagerURL" . }}#/silences/new?filter=%7B{{ range $key, $value := . }}{{ $key }}%3D%22{{ $value }}%22%2C%20{{end}}alertname%3D%22{{end}}{{ .Labels.alertname }}%22%7D|Silence>) (<{{ .GeneratorURL }}|Source>) {{ .Annotations.description }}
{{ end }}
- name: OpsGenie
opsgenie_configs:
- responders:
- name: Oncall
type: team
templates.tmpl: |-
{{ define "alertmanagerURL" }}https://alertmanager.acme.com/{{ end }}
{{ define "clusterName" }}acme-eu-west-1{{ end }}
server:
global:
external_labels:
cluster: acme
baseURL: https://prometheus.acme.com/
ingress:
enabled: true
ingressClassName: nginx
annotations:
kubernetes.io/tls-acme: "true"
hosts:
- prometheus.acme.com
tls:
- secretName: prometheus.acme.com
hosts:
- prometheus.acme.com
The prometheus alerts are not forwarded to slack. When I check the logs in the alertmanager pod, I get the following error message:
level=error ts=2022-05-23T07:16:16.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5324 err="slack/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: template: :1:23: executing \"\" at <{{template \"clusterName\" .}}>: template \"clusterName\" not defined"
The error is pretty clear and self explicit. It can't find the variable I am trying to declare. I think it's because there is no explicit path for templates.tmpl. I tried different options such as this https://github.com/helm/charts/issues/15375 but that didn't help.
To be honest I don't care whether I am declaring that variable in an external template file or in the YAML code. Whatever works.
Thanks very much for your help.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
