Load testing in mac and redhat OS throwing j.n.SocketException: Too many open files

Question

we are doing the large loading testing(50000TPS)using gatling, we are using Red Hat Enterprise Linux Server VM's for performing the testing.

Load testing is good with 500 TPS for single VM(4cpu and 8gb ram), More than 500 TPS we are not achieving the our target metrics

Running more than 3000 TPS in single machine we are facing following errors

j.n.SocketException: Too many open files

connect(..) failed: Cannot assign requested address 67328 (80.49%)

Note: I'm executing the gradle load test task from same shell where Ulimit and port ranges are increased

Steps we have tried so far

1. we have Increased the TCP local_port_range values(1024 65533) in load test vms
2. Configured the Ulimit to unlimited/higher values to higher in load test vms

kindly anyone advise us on this

Reference:

Max open files per process

https://www.ibm.com/docs/en/i/7.4?topic=utilities-ulimit

Answer 1

Try tuning network settings: https://gatling.io/docs/gatling/reference/current/core/operations/#kernel-and-network-tuning

Also is that limit 3000 TPS or users per second? Depending on injection profile Gatling may add new users every second, ignoring that previous one are still active, this way you can end with hundreds of thousands open connection. Check logs for how many active users you have, if it is way above your desired TPS, then either you need a faster network, or faster app.

Gatling can generate really high load (my record is over 300k tps from a single machine) but system under test need to be fast enough to process that requests, otherwise you end up with too many open connections.