pip on Docker image cannot find Rust - even though Rust is installed

Question

I'm trying to install some Python packages, namely tokenizers from huggingface transformers, which apparently needs Rust. So I am installing Rust on my Docker build:

FROM nikolaik/python-nodejs
USER pn
WORKDIR /home/pn/app

COPY . /home/pn/app/
RUN ls -la /home/pn/app/*
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y

ENV PATH ~/.cargo/bin:$PATH

# Install Python dependencies.
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN python load_model.py

# to be equal to the cores available.
CMD exec gunicorn --bind :$PORT --workers 4 --threads 4 app:app

But still, It seems like pip cannot find Rust when installing tokenizers:

Building wheels for collected packages: tokenizers
#11 103.2   Building wheel for tokenizers (pyproject.toml): started
#11 104.6   Building wheel for tokenizers (pyproject.toml): finished with status 'error'
#11 104.6   error: subprocess-exited-with-error
#11 104.6
#11 104.6   × Building wheel for tokenizers (pyproject.toml) did not run successfully.
#11 104.6   │ exit code: 1
#11 104.6   ╰─> [51 lines of output]
...
  error: can't find Rust compiler
#11 104.6
#11 104.6       If you are using an outdated pip version, it is possible a prebuilt wheel is available for this package but pip is not able to install from it. Installing from the wheel would avoid the need for a Rust compiler.

Why does this happen? How can I make sure Rust is availabe?

Answer 1

It seems the installer adds a line to your .bashrc file that sets up the path. .bashrc is only run if you're in an interactive shell which you aren't when you're running a build script. So your path isn't set up to include the directory with the Rust compiler.

As far as I can see, the compiler is installed in $HOME/.cargo/bin. In your case, that'd be /home/pn/.cargo/bin. To add that to the path, you can add an ENV line to your Dockerfile like this

FROM nikolaik/python-nodejs
USER pn
WORKDIR /home/pn/app

COPY . /home/pn/app/
RUN ls -la /home/pn/app/*
RUN curl --proto '=https' --tlsv1.2 -sSf -y https://sh.rustup.rs | sh
ENV PATH /home/pn/.cargo/bin:$PATH

# Install Python dependencies.
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
RUN python load_model.py

# to be equal to the cores available.
CMD exec gunicorn --bind :$PORT --workers 4 --threads 4 app:app

If that doesn't work, try running a shell in the image with the command

docker run --rm -it <image name> bash

Then you can poke around and try to find the directory the rust compiler is installed in.