Ubuntu 16 gives "fork: retry: Resource temporarily unavailable", Ubuntu 20 doesn't

Question

I have 2 similar machines in terms of HW. One has Ubuntu 16, the other Ubuntu 20.

I'm running a python program that is meant to open 30K TCP connections to an end point. The Ubuntu 20 machine machine was able to do the job well just by doing these 2 commands before executing the program:

#ulimit -n 1000000 #ulimit -u 1000000

However the Ubuntu 16 machine after creating 12K connections gives this:

-su: fork: retry: No child processes -su: fork: retry: Resource temporarily unavailable -su: fork: retry: Resource temporarily unavailable -su: fork: retry: Resource temporarily unavailable

Any idea what may be causing Ubuntu 16 to behave like that while Ubuntu 20 seems fine?

Note: I tried to do few things now from different posts but none has worked.

Thanks in advance.

Answer 1

I think the maximum number of processes overall is lower on Ubuntu 16.04 vs 20.04

i.e. On Ubuntu 16.04 /proc/sys/kernel/pid_max is 32768, while on 18.04 it's 131072

I think you might have enough other processes to hit this limit, at least it's worth checking pid_max to see.

Also it would be better to write your test program to make many connections from a single process/thread using async code, as that would allow higher limits.

PS: Ubuntu 16.04 is end-of-life (unless you're paying for the extension), so you might want to ensure you upgrade.

PS: Ubuntu versions (except the minimial-types) need to second digit block to be correct.

Answer 2

OK the solution here was to stay away from Ubuntu 16