Set up virtualenv with Paramiko SSH

Question

I have some limited experience with Python and Django in Windows, and now I am trying to understand how to deploy my code to an Ubuntu 16.04 LTS VPS. Having read various tutorials and a lot of answers on SE, I managed to proceed pretty far (well, for me), but now I am stuck.

Manually (via Putty) I can do the following:

# check that Python 3.5 is installed
python3 --version  
# install pip
sudo -kS apt-get -y install python3-pip  
# upgrade pip to newest version
pip3 install --upgrade pip
# check result
pip3 --version  
# install venv
sudo -kS pip3 install virtualenv virtualenvwrapper 
# create venv
virtualenv ~/Env/firstsite  
# make sure venv is created 
ls -l ~/Env/firstsite/bin/python  # /home/droplet/Env/firstsite/bin/python3.5 -> python3
# switch on venv
source ~/Env/firstsite/bin/activate  # (firstsite) droplet@hostname:~$
# check that python3 is taken from venv
which python3  # /home/droplet/Env/firstsite/bin/python3

So the virtual environment is properly created and switched on. I could proceed installing Django.

However when I am trying to do exactly the same in the automated regime, using Paramiko (I execute commands using paramiko.SSHClient().exec_command(cmd, input_string, get_pty=False), everything goes exactly the same way, until the last command:

exec_command('which python3')

returns /usr/bin/python3. So I assume source activate doesn't work via Paramiko's SSH.

1. Why?
2. How can I cope with it?
3. Can I check that the venv is enabled in some more direct (and reliable) way?

Answer 1

Taken from @Pablo Navarro's answer here :How to source virtualenv activate in a Bash script helped me with this same issue (activating environments in a paramiko ssh session).

In the exec_command give the path to the python executable within the environment eg:

stdin, stdout, stderr = ssh.exec_command(/path/to/env/bin/python script.py)

In my case (using miniconda and a env called pyODBC):

stdin, stdout, stderr = ssh.exec_command(~/miniconda2/envs/pyODBC/bin/python run_script.py)

running the command ~/miniconda2/envs/pyODBC/bin/python -m pip list printed the list of modules in this env to confirm

Answer 2

We can easily activate the virtualenv and execute commands on same.

Example:

import paramiko

hostname = 'host'
port = 22
username = 'root'
password = 'root'
s = paramiko.SSHClient()
s.load_system_host_keys()
s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
s.connect(hostname, port, username, password)
command = 'source /root/Envs/env/bin/activate;python3 --version;qark;echo hello'
(stdin, stdout, stderr) = s.exec_command(command)
for line in stdout.readlines():
    print(line)
for line in stderr.readlines():
    print(line)
s.close()

Answer 3

If you are using anaconda and creating your virtual environments that way, I found a work around. Taken from [this github page][1] I use send the following command to my remote pc through paramiko

f'source ~/anaconda3/etc/profile.d/conda.sh && conda activate {my_env} && {command}'

I also wish you could just activate a venv and then all the following commands would be in the venv, but this work around is nice since the only thing I have to change is the venv name. Since everythnig is in one line, it executes perfectly and I don't need to reactivate anything. If you just have a wrapper function in python it makes it all very easy to use and read. Something like this:

def venv_wrapper(command, ssh, venv=None):
    if venv:
        conda_location = 'source ~/anaconda3/etc/profile.d/conda.sh'
        activate_env = f'conda activate {venv}'
        command = f'{conda_location} && {activate_env} && {command}'
    ssh.exec_command(command, get_pty=True)

I just send all of my commands through this code (which is a little more developed/complicated in my own toolkit) whether or not im using a venv. Works pretty nicely so far [1]: https://github.com/conda/conda/issues/7980