'Joblib not working `RLIMIT_NPROC -1 current, -1 max`

I have been stuck with joblib for several days. I am trying to run in parallel some short OpenCV image processing pipeline. Apparently, there is some conflict I am unaware of and neither can I google anything about it. Unfortunately, I have troubles designing proper minimal reproducible example, but maybe someone would help me based on the error messages.

I was first getting this error message (despite the error message, the code worked):

[ERROR:[email protected]] global /io/opencv/modules/core/src/parallel_impl.cpp (240) WorkerThread 43: Can't spawn new thread: res = 11

Now, the following error has added and the code does not run at all, with maximum number of allowed processes somehow being -1:

OpenBLAS blas_thread_init: pthread_create failed for thread 38 of 45: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max

I've tried to set the number of OpenCV threads to 1 (cv2.setNumThreads(1)) so that the parallelization happens only on the side of joblib but that only changed the error:

  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/parallel.py", line 968, in __call__
    n_jobs = self._initialize_backend()
  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/parallel.py", line 736, in _initialize_backend
    **self._backend_args)
  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 470, in configure
    self._pool = MemmappingPool(n_jobs, **memmappingpool_args)
  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/pool.py", line 323, in __init__
    super(MemmappingPool, self).__init__(**poolargs)
  File "/root/.cache/pypoetry/virtualenvs/env-py3.7/lib/python3.7/site-packages/joblib/pool.py", line 205, in __init__
    super(PicklingPool, self).__init__(**poolargs)
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/pool.py", line 176, in __init__
    self._repopulate_pool()
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/pool.py", line 241, in _repopulate_pool
    w.start()
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
    return Popen(process_obj)
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/root/.pyenv/versions/3.7.12/lib/python3.7/multiprocessing/popen_fork.py", line 70, in _launch
    self.pid = os.fork()
BlockingIOError: [Errno 11] Resource temporarily unavailable

What is wrong with using OpenCV with joblib? Can it be done?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source