'"ImportError: *.so: cannot open shared object file" after pip installing extension module built with pybind11

I'm running into some peculiar behavior that I don't understand. I have a python/C++ module (call it X) built using pybind11. In my build directory I have

build
|- X
   |- a.so
   |- b.so 
   |- X.cpython-38-x86_64-linux-gnu.so (which depends on a.so and b.so)

When I'm in my build directory, python -c "import X" works, which is great.

Then, I generated a whl file for my module, and pip installed it in my virtualenv, so I have the same files above installed in ~/venv/lib/python3.8/site-packages/X. However, python -c "import X" then resulted in "ImportError: a.so: cannot open shared object file: No such file or directory", so presumably the X.cpython*.so was loaded, but the OS doesn't know where to load the dependent .so files (a.so, b.so).

As a workaround, I set LD_LIBRARY_PATH=~/venv/lib/python3.8/site-packages/X, and it resolves the issue, but I don't think that's the proper way. Can someone shed some light on why this is happening and the proper way to resolve it?

pythonc++pybind11


Solution 1:[1]

See here and references therein. In this case you will want to add -Wl,-rpath,$ORIGIN to the extra linker arguments when defining your extension e.g.

Extension(Extension(
    'X',
    sources=[ ... ]
    extra_link_args=['-Wl,-rpath,$ORIGIN', ... ]
    ... 
)

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 ChrisD

Related Questions