'Activate conda environment in docker
I need to activate environment in docker and run a command in this environment. I create the environment, but then I try to activate this environment and run the command in this way:
CMD [ "source activate mro_env && ipython kernel install --user --name=mro_env" ]
but when I ran docker I get an error:
[FATAL tini (8)] exec source activate mro_env && ipython kernel install
--user --name=mro_env failed: No such file or directory
This is the whole Dockerfile:
FROM continuumio/miniconda3
ADD /src/mro_env.yml /src/mro_env.yml
RUN conda env create -f /src/mro_env.yml
# Pull the environment name out of the mro_env.yml
RUN echo "source activate $(head -1 /src/mro_env.yml | cut -d' ' -f2)" > ~/.bashrc
ENV PATH /opt/conda/envs/$(head -1 /src/mro_env.yml | cut -d' ' -f2)/bin:$PATH
CMD [ "source activate mro_env && ipython kernel install --user --name=mro_env" ]
Solution 1:[1]
Followed this tutorial and it worked. Example Dockerfile:
FROM continuumio/miniconda
WORKDIR /usr/src/app
COPY ./ ./
RUN conda env create -f environment.yml
# Make RUN commands use the new environment:
SHELL ["conda", "run", "-n", "myenv", "/bin/bash", "-c"]
EXPOSE 5003
# The code to run when container is started:
ENTRYPOINT ["conda", "run", "-n", "myenv", "python3", "src/server.py"]
Update:
You can use "conda run --no-capture-output" to not buffer IO if you use the 4.9 version of conda. Updated Dockerfile:
FROM continuumio/miniconda
WORKDIR /usr/src/app
COPY ./ ./
RUN conda env create -f environment.yml
# Make RUN commands use the new environment:
SHELL ["conda", "run", "--no-capture-output", "-n", "myenv", "/bin/bash", "-c"]
EXPOSE 5003
# The code to run when container is started:
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "myenv", "python3", "src/server.py"]
Solution 2:[2]
You can set CONDA_DEFAULT_ENV
Like this:
FROM continuumio/miniconda3
ARG conda_env=mro_env
ADD /src/environment.yml /src/environment.yml
RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/$conda_env/bin:$PATH
ENV CONDA_DEFAULT_ENV $conda_env
CMD [ "python", "test.py" ]
UPDATE:
Better use activate. Work for me:
FROM continuumio/miniconda3
ADD /src/environment.yml /src/environment.yml
RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate mro_env"
CMD [ "python", "test.py" ]
Solution 3:[3]
For me, the solution introduced here worked seemlessly:
FROM continuumio/miniconda3
RUN conda create -n env python=3.6
RUN echo "source activate env" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH
Solution 4:[4]
As the user merv points out in one of the comments above (sorry don't have enough rep to vote up the comment) conda run buffers all stdout/stderr, thus making it not workable for applications interacting or even just displaying logs over I/O.
I noticed there was no accepted answer, so I just post what has worked very well for me:
You can use an entrypoint script to activate the conda enviroment and let it take over further input from the Dockerfile such that the python script can be executed within the activated conda environment
Example Dockerfile:
FROM continuumio/miniconda3
# make docker use bash instead of sh
SHELL ["/bin/bash", "--login", "-c"]
# copy all necessary files
COPY environment.yml .
COPY ownchain/mypyscript.py .
COPY entrypoint.sh /usr/local/bin/
# make entrypoint script executable
RUN chmod u+x /usr/local/bin/entrypoint.sh
# create environment
RUN conda env create -f environment.yml
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
CMD ["python", "mypyscript.py"]
Where the entrypoint.sh looks like this:
#!/bin/bash --login
set -e
# activate conda environment and let the following process take over
conda activate myenv
exec "$@"
All credit to David R. Pugh from this post that has more details, in particular with regards to Jupyter.
Solution 5:[5]
Here is how to run a script in a conda environment without buffering all standard input/output.
The --no-capture-output option is available since conda version 4.9.
FROM continuumio/miniconda3
COPY ./environment.yml ./environment.yml
RUN conda env create && conda clean --all -f --yes
ENTRYPOINT ["conda", "run", "--no-capture-output", "-n", "the_environment_name", "python", "myscript.py"]
Solution 6:[6]
The problem for me was that running the command conda activate env inside docker after installing caused conda to ask me to use the conda init bash command. However, this command asks you to restart the shell, which we don't want to do inside docker. So the solution is to realize that the reason conda is asking you to restart the shell is because it has modified and wants to reload the contents of ~/.bashrc. We can do this manually and forego the need for restarting the shell using:
. ~/.bashrc
Here's the full dockerfile for those who want it:
FROM ubuntu:18.04
# update apt and get miniconda
RUN apt-get update \
&& apt-get install -y wget \
&& wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
# install miniconda
ENV PATH="/root/miniconda3/bin:$PATH"
RUN mkdir /root/.conda && bash Miniconda3-latest-Linux-x86_64.sh -b
# create conda environment
RUN conda init bash \
&& . ~/.bashrc \
&& conda create --name test-env python=3.7 \
&& conda activate test-env \
&& pip install ipython
Edit - deal with comment
The commenter is correct in that the above example doesn't work for switching conda environments within docker. More recently, I figured out that to get this to work, you should start every RUN command with conda init bash && .~/.bashrc && conda activate env.
Here's another example:
FROM quay.io/pypa/manylinux2014_x86_64
RUN yum install -y wget
RUN wget https://repo.anaconda.com/miniconda/Miniconda3-py39_4.9.2-Linux-x86_64.sh
RUN bash Miniconda3-py39_4.9.2-Linux-x86_64.sh -b -p /Miniconda3
RUN /Miniconda3/bin/conda create -y --name py37 python=3.7 pytest
RUN /Miniconda3/bin/conda create -y --name py38 python=3.8 pytest
RUN /Miniconda3/bin/conda create -y --name py39 python=3.9 pytest
RUN /Miniconda3/bin/conda init && bash ~/.bashrc && . ~/.bashrc
ENV conda /Miniconda3/bin/conda
ENV bashrc /root/.bashrc
# install numpy in each env
RUN $conda init && . $bashrc && conda activate py37 && pip install numpy
RUN $conda init && . $bashrc && conda activate py38 && pip install numpy
RUN $conda init && . $bashrc && conda activate py39 && pip install numpy
Solution 7:[7]
I understand that there is no one solution fits all but this is what I have been using with my Flask applications:
FROM continuumio/miniconda3
COPY environment.yml .
RUN conda env create -f environment.yml
COPY app /app
WORKDIR /app
CMD ["conda", "run", "-n", "my_env", "python", "app.py"]
The logic is very simple. First, environment file is copied followed by the creation of the environment. Next, application files (this is where all my Flask application files are located) are copied. Finally, using CMD the application is started by pointing to the environment.
This is the project directory structure I used with the Dockerfile:
-- project
-- app
-- app.py
-- environment.yaml
-- Dockerfile
Note that this does not activate the environment per se but points to it at run time in the CMD command
Solution 8:[8]
I stumbled upon this question while trying to activate an environment and then installing some packages inside it. The SHELL solution did not work for me (maybe because I was attempting this on an older version of conda - 4.5.4 to be precise).
The solution is to activate the environment and run all required commands inside a new shell. Also, remember that each RUN will start a new shell that will not remember anything from the previous RUN.
FROM continuumio/miniconda3
ADD /src/environment.yml /src/environment.yml
RUN conda env create -f /src/environment.yml
ENV PATH /opt/conda/envs/mro_env/bin:$PATH
RUN /bin/bash -c "source activate mro_env \
&& conda config --add channels conda-forge \
&& conda install Jupiter \
&& conda env list"
CMD [ "python", "test.py" ]
Note every command is within "" of the same /bin/bash -c.
Solution 9:[9]
Similar to other answers but with SHELL which looks more cleaner
RUN wget \
https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
&& mkdir /root/.conda \
&& bash Miniconda3-latest-Linux-x86_64.sh -b \
&& rm -f Miniconda3-latest-Linux-x86_64.sh \
&& echo "source activate base" > ~/.bashrc
SHELL ["/bin/bash", "-c"]
ARG CONDA_ENV=env_name
# Create env
RUN conda --version \
&& conda create -n ${CONDA_ENV} python=3.7 \
&& source activate ${CONDA_ENV}
Solution 10:[10]
If you don't need to change environments away from the base you could also do this:
COPY conda.yaml /
RUN { echo "name: base"; tail +2 /conda.yaml; } > /base.yaml
RUN conda env update --file /base.yaml --prune
The environment in conda.yaml could have any name since we replace it with base.
Solution 11:[11]
Since, conda run is an experimental feature the correct way is to add this line to your Dockerfile
SHELL [ "/bin/bash", "--login", "-c" ]
after this you can continue with
RUN conda init bash
and then continue to activate your environment with
RUN conda activate <env_name>
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow