'How to query container memory limit in Prometheus

I am using Prometheus tool for monitoring my Kubernetes cluster.

I have set a resource limit(memory limit) in my deployments and need to configure a panel for showing the total memory available. Please let me know the query needed to run in Prometheus for getting the total memory limit available for my deployment.

kubernetesprometheusprometheus-operator


Solution 1:[1]

It is possible using metrics kube_pod_container_resource_limits_memory_bytes (provided by kube-state-metrics) and container_memory_usage_bytes (provided by kubelet/cAdvisor)

label_replace(
  label_replace(
    kube_pod_container_resource_limits_memory_bytes{},
    "pod_name", 
    "$1", 
    "pod", 
    "(.+)"
  ),
  "container_name", 
  "$1", 
  "container", 
  "(.+)"
) 
-
on(pod_name,namespace,container_name) 
avg(
      container_memory_usage_bytes{pod_name=~".+"}
)
by (pod_name,namespace,container_name)

A little explanation of the query: It is a subtraction of the memory limit and the actual usage. label_replace functions are needed to match the label names of both metrics, as they are obtained from different targets. avg is used to get the average between pod restarts, as every pod restart creates a new metric. {pod_name=~".+"} is used to filter metrics from container_memory_usage_bytes that are not useful for this case

Solution 2:[2]

The following PromQL query returns per-container free memory starting from Kubernetes v1.16:

kube_pod_container_resource_limits{resource="memory"}
  - on(namespace, pod, container)
container_memory_usage_bytes

The on() modifier instructs Prometheus to take into account only the listed labels when finding pairs of time series with identical labels on the left and the right side of - operator. See these docs for details.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Ignacio Millán
Solution 2 valyala

Related Questions