Ulimit not having effect in ssh session

Question

I am building a docker image and I'd like to increase the maximum amount of files that can be opened. I tried several things but none of them worked when I opened a new SSH session that connected to the container. They did work when executing a bash into the container.

I tried, in the docker build:

RUN echo "DefaultLimitNOFILE=65535" >> /etc/systemd/system.conf

Also tried:

RUN set ulimit -n 65535
RUN set ulimit -Sn 65535
RUN set ulimit -Hn 65535

I tried to add --ulimit nofile=65535:65535 both to the docker run and docker build command.

After I start the image and I log into it through SSH, the soft limit is never the one I set.

Docker build:

FROM nvcr.io/nvidia/deepstream:6.0-triton

ENV GIT_SSL_NO_VERIFY=1

# SETUP PYTHON
RUN sh docker_python_setup.sh
RUN update-alternatives --set python3 /usr/bin/python3.8
RUN apt install --fix-broken -y
RUN apt -y install python3-gi python3-gst-1.0 python-gi-dev git python3 python3-pip cmake g++ build-essential \
  libglib2.0-dev python3-dev python3.8-dev libglib2.0-dev-bin python-gi-dev libtool m4 autoconf automake

# DEEPSTREAM PYTHON BINDINGS
RUN cd /opt/nvidia/deepstream/deepstream-6.0/sources/apps && \
    git clone https://github.com/NVIDIA-AI-IOT/deepstream_python_apps.git
RUN cd /opt/nvidia/deepstream/deepstream-6.0/sources/apps/deepstream_python_apps && \
    git submodule update --init
RUN cd /opt/nvidia/deepstream/deepstream-6.0/sources/apps/deepstream_python_apps/3rdparty/gst-python/ && \
   ./autogen.sh && \
   make && \
   make install

RUN pip3 install --upgrade pip
RUN cd /opt/nvidia/deepstream/deepstream-6.0/sources/apps/deepstream_python_apps/bindings && \
    mkdir build && \
    cd build && \
    cmake -DPYTHON_MAJOR_VERSION=3 -DPYTHON_MINOR_VERSION=8 -DPIP_PLATFORM=linux_x86_64 -DDS_PATH=/opt/nvidia/deepstream/deepstream-6.0 .. && \
    make && \
    pip3 install pyds-1.1.0-py3-none-linux_x86_64.whl
RUN cd /opt/nvidia/deepstream/deepstream-6.0/sources/apps/deepstream_python_apps && \
    mv apps/* ./

# RTSP DEPENDENCIES
RUN apt update && \
    apt install -y python3-gi python3-dev python3-gst-1.0
RUN apt update && \
    apt install -y libgstrtspserver-1.0-0 gstreamer1.0-rtsp && \
    apt install -y libgirepository1.0-dev && \
    apt-get install -y gobject-introspection gir1.2-gst-rtsp-server-1.0

# DEVELOPMENT AND DEBUGGING TOOLS
RUN apt install -y ipython3 graphviz graphviz-dev ffmpeg

# SSH AND REMOTE LOGIN FOR DEVELOPMENT PURPOSES
RUN apt update && apt install -y openssh-server
RUN mkdir /var/run/sshd
RUN echo 'root:230idsjfjzJNJK3' | chpasswd
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
# SSH login fix. Otherwise user is kicked off after login
RUN sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
ENV NOTVISIBLE "in users profile"
RUN echo "export VISIBLE=now" >> /etc/profile
RUN sed -i 's/\(^Port\)/#\1/' /etc/ssh/sshd_config && echo Port 2222 >> /etc/ssh/sshd_config
# Export 2222 for SSH server
EXPOSE 2222

# SET ULIMIT USING THE COMMANDS ABOVE ....

# STARTUP
# Disable previous entrypoint.
ENTRYPOINT []
# Set default dir
WORKDIR /src
# Enable SSH for debug on remote server
CMD ["/usr/sbin/sshd", "-D"]

In the SSH session I always get the value:

root@ip-x-x-x-x:~# ulimit -n
1024
root@ip-x-x-x-x:~# ulimit -Sn
1024
root@ip-x-x-x-x:~# ulimit -Hn
1048576

I'd like to set the limit for all future SSH sessions.

EDIT: I noticed if I open a shell into the container, the soft limit is actually equal to the hard limit even without specifying anything. So the default limit is 1048576. But if I open an SSH session into the container the soft limit is 1024. How can I solve this?

Answer 1

You should also use prlimit and update the value of the current session (Bash) you are in. Try running the below script.

    echo "add openfiles limit..........................."
    sudo cp /etc/security/limits.conf  /etc/security/orig_limits.conf
    sudo cat <<EOT >> /etc/security/limits.conf
    *               hard    nofile          33000 
    *               soft    nofile          33000
    root            hard    nofile          33000 
    root            soft    nofile          33000
    EOT
    
    sudo echo "session required pam_limits.so" > /etc/pam.d/common-session
    sudo ulimit -n 33000
    ulimit -u unlimited
    
    update_ulimit_per_pid(){
        sudo echo "prlimit for pid "$pid" before updating is "$(ulimit -n)
        sudo echo "Updating ulimit for pid: "$pid
        sudo prlimit --pid $pid --nofile=33000:33000
        sudo echo "prlimit for pid "$pid" after updating is "$(ulimit -n)
    }
    
    for pid in `ps -ef | grep 'bash' | awk '{print $2}'` ; do update_ulimit_per_pid ; done

This should work. This will not only update ulimit when you relogin, but also the in the bash session you are in.