'prometheus with node_exporter sidecar openshift

I tried to deploy prometheus pod with node_exporter sidecar but when I use the volumeMounts on sidecar node_exporter container it gives me "Multi-attach error" cannot mount or no bound. I want to use node-exporter belong with all my pods/apps in order to have usage disk metrics, in openshift we don't have any pvc disk usage monitoring metrics and is very annoying.

my yaml.config:

 containers:
    - args:
        - '--config.file=/etc/prometheus/prometheus.yml'
        - '--web.enable-lifecycle'
      command:
        - /bin/prometheus
      image: ${DOCKER_REPOSITORY}/prom/prometheus:v2.0.0
      imagePullPolicy: IfNotPresent
      name: prometheus
      ports:
        - containerPort: 9090
          protocol: TCP
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      volumeMounts:
        - mountPath: /prometheus
          name: prometheus-data
        - mountPath: /etc/prometheus
          name: prometheus-configmap
      image: quay.io/prometheus/node-exporter
      imagePullPolicy: IfNotPresent
      name: node-exporter-sidecar
      ports:
        - containerPort: 9100
          protocol: TCP
      volumeMounts:
        - mountPath: /prometheus
          name: prometheus-data
     - kind: PersistentVolumeClaim
       apiVersion: v1
       metadata:
         name: prometheus-ls-data
       spec:
         accessModes: [ "ReadWriteOnce" ]
         resources:
           requests:
             storage: "${PV_SIZE}"
         storageClassName: "cinder-performant"


Solution 1:[1]

Node exporter is deployed as part of default openshift monitoring stack and its lifecycle is managed by cluster-monitoring-operator. It is deployed as a daemonset and runs as pod on each node to extract its metrics.

Have you installed or deployed default prometheus monitoring stack in your cluster if not you just need to deploy the default stack and node exporter will be installed as part of that.

Solution 2:[2]

Following Prometheus Metrics are exposed as part of default openshift monitoring stack ( as Kubelet Target ) . you can use this to calculate the usage of your pvs .

kubelet_volume_stats_available_bytes kubelet_volume_stats_capacity_bytes kubelet_volume_stats_inodes kubelet_volume_stats_inodes_used

Cant you see the above metrics in your prometheus instance.

Solution 3:[3]

In openshift 3.9 volume metrics are exposed by prometheus as below

enter image description here

So search for these metrics.

Solution 4:[4]

enter image description here

Please see the volume metrics i have in the cluster from the Prometheus

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Manmohan Mittal
Solution 2 Manmohan Mittal
Solution 3 Manmohan Mittal
Solution 4 Petrisor Petrisor