Building a package referencing a package in a parent directory in setup.py

Question

I am trying to build a pip package from source code in a Git repository that has multiple packages that share a common package. (I am not allowed to change the structure of this Git repository.)

The structure is:

├── common
│   ├── __init__.py
│   ├── run_helpers
│   │   ├── __init__.py
│   │   ├── aws.py
│   │   └── s3.py
└── components
    └── redshift_unload
        ├── redshift_unload
        │   ├── __init__.py
        │   └── run.py
        └── setup.py

My setup.py is as follows:

from setuptools import setup, find_packages

setup(
    ...
    packages=find_packages(),
    package_dir={"": "."},
    entry_points={
        "console_scripts": ["redshift_unload=redshift_unload.run:main"]
    }
)

Looking at other answers here, things I have tried so far include:

• Specifying the actual package names in the packages= line instead of using find_packages().
• Passing where="../../" to find_packages()
• Using find_packages() + find_packages(where="../../") in the packages=` line.
• Everything I can think of in the packages_dir line.

When I run pip install . I get, the package installs fine, but then when I run the installed python script I get:

# redshift_unload
Traceback (most recent call last):
  File "/usr/local/bin/redshift_unload", line 5, in <module>
    from redshift_unload.run import main
  File "/usr/local/lib/python3.8/site-packages/redshift_unload/run.py", line 9, in <module>
    from common._run_helpers.aws import get_boto3_session
ModuleNotFoundError: No module named 'common'

What did work:

• If I moved the common directory to components/redshift_unload, then it works fine. But I can't do this. I also tried placing a symlink there in its place, but seems like that doesn't work either.

Is there a way to make this work?

Answer 1

I believe I have found the best solution to this.

Based on this comment here, I concluded that what I am trying to do is not intended or supported.

However, I found a workaround as follows works fine:

from pathlib import Path
from shutil import rmtree, copytree

from setuptools import setup, find_packages

src_path = Path(os.environ["PWD"], "../../common")
dst_path = Path("./common")

copytree(src_path, dst_path)

setup(
    ...
    packages=find_packages(),
    package_dir={"": "."},
    entry_points={
        "console_scripts": ["redshift_unload=redshift_unload.run:main"]
    }
)

rmtree(dst_path)

The key insight here is that, while packaging occurs in a temporary directory, the value of os.environ["PWD"] is available to the process, such that the common directory can be copied temporarily and then cleaned up again (using shutil functions copytree and rmtree) into a location that will be found by find_packages() before the setup() function is called.