pip-installed package inside conda virtual environment is not isolated

Question

I am developing a simple python package (on macOS 10.14) and I have problems in setting the instructions how to install it. Right now the package is not available anywhere yet, but think of it as a dummy "hello world" package with a dummy "hello world" function inside a dummy "hello world" module. Of course it has a proper setup.py script that would allow users to install and uninstall the package with pip.
When I install and test it myself everything works fine, the problem is not related to the package itself.
The issue is that I cannot make conda virtual environments and pip work together... Next to my setup.py script there is a environment.yaml file that specifies the dependancies required for my package. Based on this file I create a virtual environment with:

conda env create --prefix ENV -f environment.yaml

I have conda 4.7.12 with Python 3.7.3 inside so the virtual environment has it's own pip. So I activate the virtual environment and explicitly call the pip inside to install my package in the virtual environment:

/Users/my_name/Desktop/dev_dir/ENV/bin/pip install . --user

The installation is successful and the package can be imported. However when I deactivate the virtual environment with conda deactivate and run python interpreter from the conda base environment (version 3.6.9) I can still load my package! For some reason it is available outside of that particular virtual environment...

Later, when I run the 'inner' pip from conda base shell:

/Users/my_name/Desktop/dev_dir/ENV/bin/pip uninstall pkg

The removal seems to go through as well. I get a message:

Uninstalling pkg-0.0.0.9000:
  Would remove:
    /Users/my_name/.local/lib/python3.7/site-packages/pkg-0.0.0.9000.dist-info/*
    /Users/my_name/.local/lib/python3.7/site-packages/pkg/*
Proceed (y/n)? y
  Successfully uninstalled pkg-0.0.0.9000

suggesting that the package was indeed installed in a directory .local, outside conda virtual environments.

And the best for the last: even after this uninstallation when I run python interpreters (regardless of which environment from) and I try to import pkg it still works! when I then type pkg in the interpreter I get the path to my development directory:

>>> import pkg
>>> pkg
<module 'pkg' from '/Users/my_name/Desktop/dev_dir/pkg/__init__.py'>

Could someone please help me disentangle this mess? I would like to have my package installed inside the virtual environment, nicely isolated. And also - it should be gone after uninstallation, right?

PS. PYTHONPATH variable is never set anywhere at any stage, I have been checking that...

Answer 1

@FlyingTeller already correctly identified the issue. I just wanted to point out that you could further streamline your process by adding the installation for your package into your YAML definition. For example,

name: my_env
channels:
 - defaults
dependencies:
 - python=3.7.3
 - pip
 - pip:
   - -e /Users/my_name/Desktop/dev_dir/pkg

This is also further in line with the best practices (see "Using Pip in a Conda Environment").

Answer 2

Just wanted to hopefully clear some up by telling you this keeps happen to many and if you forget the rule that is NO root install with conda, all rules for your files might change and suddenly it keeps asking for sudo AND fails. Conda = NO SUDO! Hope you got it fixed!

Answer 3

You have to add the pip package to your environment (see https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html), otherwise the packages will be installed by the global pip installation such that those packages can be accessed by all environments. Therefore, create an environment using
conda create --name exampleenv pip
instead of
conda create --name exampleenv