'define variable in alertmanager configuration yaml file to forward prometheus alerts to slack

I am stuck with a basic YAML problem but I am not sure what's the proper way around it and YAML isn't my strong suit.

I am trying to get the below code to work. The "slack_configs:" section is where I get an error. Whatever I put under the "templates.tmpl" section does not seem to be picked up.

Here's the code:

alertmanager:
    baseURL: https://alertmanager.acme.com/
    ingress:
        enabled: "true"
        ingressClassName: nginx
        annotations:
            kubernetes.io/tls-acme: "true"
        hosts:
            - alertmanager.acme.com
        tls:
            - secretName: alertmanager.acme.com
              hosts:
                - alertmanager.acme.com
## alertmanager ConfigMap entries
##
alertmanagerFiles:
    alertmanager.yml:
        global:
            slack_api_url: https://hooks.slack.com/services/ACME
            opsgenie_api_key: 1234
        route:
            group_wait: 10s
            group_interval: 1m
            repeat_interval: 2h
            receiver: slack
            routes:
                - receiver: slack
                  repeat_interval: 1m
                  continue: true
                  continue: true
                - receiver: OpsGenie
                  repeat_interval: 1m
        receivers:
            - name: slack
              slack_configs:
                - channel: '#alertmanager'
                  send_resolved: true
                  title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}{{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .alertname }}{{ end }}{{ end }}'
                  text: |-
                    *Cluster:* {{ template "clusterName" . }}
                    {{ range .Alerts }}•  (<{{ with .Labels }}{{template "alertmanagerURL" . }}#/silences/new?filter=%7B{{ range $key, $value := . }}{{ $key }}%3D%22{{ $value }}%22%2C%20{{end}}alertname%3D%22{{end}}{{ .Labels.alertname }}%22%7D|Silence>) (<{{ .GeneratorURL }}|Source>) {{ .Annotations.description }}
                    {{ end }}
            - name: OpsGenie
              opsgenie_configs:
                - responders:
                    - name: Oncall
                      type: team
    templates.tmpl: |-
        {{ define "alertmanagerURL" }}https://alertmanager.acme.com/{{ end }}
        {{ define "clusterName" }}acme-eu-west-1{{ end }}
server:
    global:
        external_labels:
            cluster: acme
    baseURL: https://prometheus.acme.com/
    ingress:
        enabled: true
        ingressClassName: nginx
        annotations:
            kubernetes.io/tls-acme: "true"
        hosts:
            - prometheus.acme.com
        tls:
            - secretName: prometheus.acme.com
              hosts:
                - prometheus.acme.com

The prometheus alerts are not forwarded to slack. When I check the logs in the alertmanager pod, I get the following error message:

level=error ts=2022-05-23T07:16:16.547Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=5324 err="slack/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: template: :1:23: executing \"\" at <{{template \"clusterName\" .}}>: template \"clusterName\" not defined"

The error is pretty clear and self explicit. It can't find the variable I am trying to declare. I think it's because there is no explicit path for templates.tmpl. I tried different options such as this https://github.com/helm/charts/issues/15375 but that didn't help.

To be honest I don't care whether I am declaring that variable in an external template file or in the YAML code. Whatever works.

Thanks very much for your help.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source