Python in Anaconda packages different from command line to jupyterhub (linux)

Question

Strange problem here. I am working on Ubuntu 20.04 using Anaconda to set up some python environments. Specifically I am trying to set up tensorflow to use my GPU but that is just a detail. The problem is that everything works correctly when I open up a terminal and execute the following commands (for environment called tf):

conda activate tf
python

Then in the python prompt I run

import tensorflow as tf
tf.config.list_physical_devices()

The output is good, it shows I have a GPU:

[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

So then I fire up Jupyterlab, for which I have added this environment, called tf, and I run the same commands, and it says it can't find some packages and only shows CPU as physical devices. This is the problem. Here is the output:

2022-02-16 09:20:03.656417: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-02-16 09:20:03.656737: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-02-16 09:20:03.656836: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2022-02-16 09:20:03.656927: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2022-02-16 09:20:03.658936: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusolver.so.11'; dlerror: libcusolver.so.11: cannot open shared object file: No such file or directory
2022-02-16 09:20:03.659034: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2022-02-16 09:20:03.659198: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
[PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]

The same thing happens if I open up a console using this environment. But in jupyter if I open up a regular terminal and activate tf then it works correctly.

Why is the behavior different here? This is the first time I've seen this happen. Does anyone know how to get jupyter to behave the same way the command line is behaving?

Thanks.

UPDATE I thought the problem was maybe the $PATH variable, but I enforced the path in jupyter to be the same as what is showing in the terminal session that works, but still get the same error.

Answer 1

The environment in which your Jupyter notebook is running has a wrong LD_LIBRARY_PATH, then it doesn't find the CUDA or cudnn libraries needed to use the GPU.