'Debugging Memory Leaks in a Containerised Node Application

We have Kafka cluster and on the first Kafka we installed the Kafka exporter

From dmesg we saw the following exception

[68026.617081] Memory cgroup out of memory: Kill process 27596 (kafka_exporter) score 1002 or sacrifice child
[68026.617086] Killed process 2295 (kafka_exporter) total-vm:4125008kB, anon-rss:4107164kB, file-rss:4224kB, shmem-rss:0kB

And from docker ps , we saw the kafka exporter container restart every 10 min

The yml file of the container looks like this

version: '2.4'
services:
  kafka-exporter:
    mem_limit: "4012m"
    image: kafka-exporter:v1.2.0
    restart: always
    network_mode: host
    container_name: kafka-exporter

so the behavior is like this

once kafka exporter restart then , the consuming of the kafka exporter increase until the high of the mem_linit , once kafka exporter reached the limit only then we get restart

the strange thing is that kafka exporter not really need 4G , on other machine we configured much less mem_limit and kafka exporter works fine

so we not understand why kafka exporter container reached until 4G

more info

docker --debug info
Client:
 Debug Mode: true

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 3.10.0-957.el7.x86_64
 Operating System: Red Hat Enterprise Linux Server 7.6 (Maipo)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 11.58GiB
 ID: AGBU:7ZAU:ADDN:PM5W:73GR:HKFJ:DP3D:B7A6:JVUY:EFT7:WP3V:75P3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Registry Mirrors:
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

references:

https://access.redhat.com/solutions/6128771 https://newrelic.zendesk.com/hc/en-us/articles/360059501693-RHEL7-RHEL8-DockerCE-cgroup-kernel-memory-leak https://www.ibm.com/docs/en/cloud-private/3.1.1?topic=ts-changing-cgroup-driver-systemd-red-hat-enterprise-linux docker change cgroup driver to systemd

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'Debugging Memory Leaks in a Containerised Node Application

Sources

Related Questions