'Prometheus Operator not scraping colocated etcd metrics

I have a K8s cluster with colocated etcd deployed on-prem servers, using Kubespray. I don't see the etcd metrics getting scraped by Prometheus operator. Prometheus operator deployed using helm v3.5.4. K8s version 1.22 , Helm chart prometheus-community/kube-prometheus-stack version 25.0.0 , 3 node control plane on CentOS 7.

Prometheus config shows a job for etcd - job_name: serviceMonitor/monitoring/kube-prometheus-kube-prome-kube-etcd/0 . But there is no service for etcd in the list of Services for Prometheus.

There are no endpoints defined for etcd

Missing endpoints for etcd

Values.yml (updated with volumes ) for helm deployment

prometheus:
    service: 
        type: NodePort
        externalTrafficPolicy: Local
    ingress:
        enabled: true
        annotations:
            kubernetes.io/ingress.class: "custom"
        hosts:
            - prometheus.{{ cluster_domain }}.mydomain.com
        paths: 
            - /
        pathType: Prefix
        tls:
            - secretName: 
    prometheusSpec:
        storageSpec:
            volumeClaimTemplate:
                spec:
                    accessModes: ["ReadWriteOnce"]
                    storageClassName: rook-ceph-block
                    resources:
                        requests:
                            storage: {{ monitoring.storage_size }}
        volumeMounts:
        - name: cert-vol
          mountPath: "/etc/prometheus/secrets/etcd-certs"
          readOnly: true

        volumes:
        - name: cert-vol
          secret: 
            secretName: etcd-certs

kubeEtcd:
    enabled: true
    endpoints:
      - 172.1.1.1
      - 172.1.1.2
      - 172.1.1.3
  
    service:
        port: 2379
        targetPort: 2379
    serviceMonitor:
        scheme: https
        insecureSkipVerify: true
        caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
        certFile: /etc/prometheus/secrets/etcd-certs/client.crt
        keyFile: /etc/prometheus/secrets/etcd-certs/client.key
kubernetesprometheusprometheus-operatorhelm3kube-prometheus-stack


Solution 1:[1]

I added the endpoints to kubeEtcd section to get it to work. The updated values.yaml is like below (changed IP adresses):

prometheus:
    service: 
        type: NodePort
        externalTrafficPolicy: Local
    ingress:
        enabled: true
        annotations:
            kubernetes.io/ingress.class: "custom"
        hosts:
            - prometheus.{{ cluster_domain }}.mydomain.com
        paths: 
            - /
        pathType: Prefix
        tls:
            - secretName: 
    prometheusSpec:
        storageSpec:
            volumeClaimTemplate:
                spec:
                    accessModes: ["ReadWriteOnce"]
                    storageClassName: rook-ceph-block
                    resources:
                        requests:
                            storage: {{ monitoring.storage_size }}
        volumeMounts:
        - name: cert-vol
          mountPath: "/etc/prometheus/secrets/etcd-certs"
          readOnly: true

        volumes:
        - name: cert-vol
          secret: 
            secretName: etcd-certs

kubeEtcd:
    enabled: true
    endpoints:
      - 172.1.1.1
      - 172.1.1.2
      - 172.1.1.3
  
    service:
        port: 2379
        targetPort: 2379
    serviceMonitor:
        scheme: https
        insecureSkipVerify: true
        caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
        certFile: /etc/prometheus/secrets/etcd-certs/client.crt
        keyFile: /etc/prometheus/secrets/etcd-certs/client.key

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Miko?aj G?odziak

Related Questions