Using Ray Multiprocessing on Docker

Question

I am trying to parallelize processes using Ray in a docker container.

from ray.util.multiprocessing import Pool

with Pool(self.n_cores) as pool:
   pool.starmap(func=func, iterable=list_of_parameters)

while it is perfectly working locally, when it gets run in the docker container, the following error occurs:

✗ failed cod_simulation-run Job 'cod_simulation-run' failed: | AttributeError: 'StreamCapture' object has no attribute 'fileno' Write "details cod_simulation-run" to inspect the error. Processed 1 jobs (1 failed).

I was previously performing the same thing with python multiprocessing:

import multiprocessing as mp

with mp.Pool(self.n_cores) as pool:
   pool.starmap(func=func, iterable=list_of_parameters)

and this worked both locally and in the docker container. But for efficiency reasons, I would prefer to stick to Ray.

FROM python:3.9

WORKDIR /project_name
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

RUN find .

ENV DISABLE_CONTRACTS=1

RUN pipdeptree
RUN python setup.py develop --no-deps

RUN cod-demo --help

CMD ["cod-demo"]

This is my DockerFile and I am installing ray as a requirement.

Thanks in advance for any suggestion

Answer 1

After pulling the appropriate tag based on your needs from their repo on dockerhub. You can simply run tests with:

docker run --shm-size=<shm-size> -t -i rayproject/ray

or

docker run --shm-size=<shm-size> -t -i --gpus all rayproject/ray:<ray-version>-gpu

In case you're using GPU. If you need some additional packages which aren't pre-installed on images you must install them through the terminal inside the image or fabricate a new docker-file with the downloaded image as the base image (the FROM command argument).

According to what was mentioned here!