Nginx memory leak issue

Question

We are facing issue with nginx memory leak it seems. Setup: Nginx running as deployment in GKE Nginx version 1.20.2 Nginx is used to stream HLS. We write chunk file to a google filestore(NFS service). It is mounted on /var/www/html/.

Nginx never ever recovers memory it just grows on increasing. Nginx confiuration

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Default.conf

server {
    listen       80;
    server_name  localhost;
    proxy_buffering off;
    proxy_buffer_size 4k;
    proxy_buffers 64 4k;
    proxy_busy_buffers_size 16k;
    proxy_cache_valid  200 302  1m;
    proxy_cache_valid  404  60m;
    proxy_cache_use_stale  error timeout invalid_header updating;
    proxy_redirect off;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
        add_header 'Access-Control-Allow-Origin' '*';
        add_header 'Access-Control-Allow-Credentials' 'true';
        add_header 'Access-Control-Allow-Methods' '*';
        add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
        add_header 'Access-Control-Max-Age' 1728000;


    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
    location /stub_status {
        stub_status on;
        access_log  on;
        allow all;
    }
  }
}

Screenshot of our internal monitoring system

Answer 1

First of all, what does your chart show? Memory usage of nginx worker processes? Or memory utilization of whole system?

In case of nginx memory growth it may relate to known issue (basically with OpenSSL either), see https://trac.nginx.org/nginx/ticket/2316

So either try to apply patch suggested by Maxim in that issue, or try the workaround he suggested in the last comment or upgrade to newer version of OpenSSL (PKCS11 engine) or even nginx (especially if it is linked statically).
There are enough OpenSSL-related leak issues, see also for example https://github.com/kubernetes/ingress-nginx/issues/7647 or linked within. So to veryfy it is not affected by OpenSSL, try to test it without SSL/TLS/https and check whether you'd see growth of memory usage.

Although I don't see any memory leak trying vanilla nginx 1.20.2 (without any patch, built with OpenSSL 1.1.1k) testing similar configuration (I don't see proxy_pass directive in your config so I was simply proxying to http/https upstream too). No leak reproducible at all.

In case of high system memory usage, it may be common OS cache or even some buffering of NFS, see https://askubuntu.com/a/1393696/1384131 for similar question.