'TensorFlow GPU doesn't recognize GPU dlls
I followed the following website to install the necessary GPU Support for Tensorflow 2.4.1: https://www.tensorflow.org/install/gpu followed by the actual Tensorflow installation: https://www.tensorflow.org/install/pip. I then run a sample snippet of code which should check the GPU and try running a tensor. It says it cannot see certain CUDA files, of which I pasted all of the files below (that also match with the path). I pasted all of the relevant information I could think of below.
I checked to make sure all of the dll's were created properly:
PS C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin> ls
Directory: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 2/13/2021 10:04 PM crt
-a---- 7/23/2020 6:58 AM 203776 bin2c.exe
-ar--- 7/27/2020 4:11 PM 58 compute-sanitizer.bat
-a---- 8/4/2020 1:57 AM 89057280 cublas64_11.dll
-a---- 8/4/2020 1:57 AM 167523328 cublasLt64_11.dll
-a---- 7/23/2020 6:58 AM 373248 cuda-memcheck.exe
-a---- 7/23/2020 6:58 AM 4604928 cudafe++.exe
-a---- 7/23/2020 6:59 AM 335872 cudart32_110.dll
-a---- 7/23/2020 6:59 AM 401408 cudart64_110.dll
-a---- 2/13/2021 8:49 PM 222720 cudnn64_8.dll
-a---- 2/13/2021 8:49 PM 99908096 cudnn_adv_infer64_8.dll
-a---- 2/13/2021 8:49 PM 65183232 cudnn_adv_train64_8.dll
-a---- 2/13/2021 8:49 PM 508834304 cudnn_cnn_infer64_8.dll
-a---- 2/13/2021 8:49 PM 51145728 cudnn_cnn_train64_8.dll
-a---- 2/13/2021 8:49 PM 240333312 cudnn_ops_infer64_8.dll
-a---- 2/13/2021 8:49 PM 25651712 cudnn_ops_train64_8.dll
-a---- 7/23/2020 6:59 AM 162878976 cufft64_10.dll
-a---- 7/23/2020 6:59 AM 272896 cufftw64_10.dll
-a---- 7/23/2020 6:59 AM 1305600 cuinj64_110.dll
-a---- 7/23/2020 6:58 AM 2828800 cuobjdump.exe
-a---- 7/23/2020 6:59 AM 52446720 curand64_10.dll
-a---- 7/23/2020 6:59 AM 525173248 cusolver64_10.dll
-a---- 7/23/2020 6:58 AM 314601984 cusolverMg64_10.dll
-a---- 7/23/2020 6:59 AM 159789568 cusparse64_11.dll
-a---- 7/23/2020 6:58 AM 302080 fatbinary.exe
-a---- 7/23/2020 6:59 AM 232448 nppc64_11.dll
-a---- 7/23/2020 6:59 AM 10907648 nppial64_11.dll
-a---- 7/23/2020 6:59 AM 4569600 nppicc64_11.dll
-a---- 7/23/2020 6:59 AM 7840256 nppidei64_11.dll
-a---- 7/23/2020 6:59 AM 51866112 nppif64_11.dll
-a---- 7/23/2020 6:59 AM 26065920 nppig64_11.dll
-a---- 7/23/2020 6:59 AM 6411264 nppim64_11.dll
-a---- 7/23/2020 6:59 AM 20231168 nppist64_11.dll
-a---- 7/23/2020 6:59 AM 203264 nppisu64_11.dll
-a---- 7/23/2020 6:59 AM 2856960 nppitc64_11.dll
-a---- 7/23/2020 6:59 AM 8873472 npps64_11.dll
-a---- 8/4/2020 1:57 AM 299520 nvblas64_11.dll
-a---- 7/23/2020 6:58 AM 408576 nvcc.exe
-a---- 7/23/2020 6:58 AM 334 nvcc.profile
-a---- 7/23/2020 6:58 AM 28080128 nvdisasm.exe
-a---- 7/23/2020 6:58 AM 3133952 nvjpeg64_11.dll
-a---- 7/23/2020 6:58 AM 7523328 nvlink.exe
-a---- 7/23/2020 6:58 AM 2173440 nvprof.exe
-a---- 7/23/2020 6:58 AM 225792 nvprune.exe
-a---- 7/23/2020 6:58 AM 5118976 nvrtc-builtins64_110.dll
-a---- 7/23/2020 6:58 AM 16161792 nvrtc64_110_0.dll
-a---- 7/23/2020 6:58 AM 53 nvvp.bat
-a---- 7/23/2020 6:58 AM 7407104 ptxas.exe
Here is my path:
C:\Users\jnels>echo %PATH% C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin;C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\libnvvp;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\Nsight Compute 2020.1.2;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\jnels\AppData\Local\Microsoft\WindowsApps;
I then followed the instructions for running a sample code which will test the GPU & Tensorflow to see if it works. I should expect to see a tensor returned:
PS C:\Users\jnels> python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"
2021-02-13 23:24:28.372069: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-02-13 23:24:28.372201: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2021-02-13 23:24:29.637123: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-13 23:24:29.638344: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll
2021-02-13 23:24:29.692627: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0b:00.0 name: GeForce RTX 3080 computeCapability: 8.6
coreClock: 1.71GHz coreCount: 68 deviceMemorySize: 10.00GiB deviceMemoryBandwidth: 707.88GiB/s
2021-02-13 23:24:29.692892: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2021-02-13 23:24:29.694758: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2021-02-13 23:24:29.695158: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2021-02-13 23:24:29.695609: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2021-02-13 23:24:29.696010: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2021-02-13 23:24:29.696368: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusolver64_10.dll'; dlerror: cusolver64_10.dll not found
2021-02-13 23:24:29.696722: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2021-02-13 23:24:29.697079: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2021-02-13 23:24:29.697327: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-02-13 23:24:29.698128: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-02-13 23:24:29.699113: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-13 23:24:29.699192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2021-02-13 23:24:29.699571: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
tf.Tensor(1250.3312, shape=(), dtype=float32)
Solution 1:[1]
You have not added all required installation directories to your path environment variables to support tensorflow_gpu.
Add the CUDA®, CUPTI, and cuDNN installation directories to the %PATH% environmental variable as below and update your %PATH% to match:
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\bin;%PATH%
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\extras\CUPTI\lib64;%PATH%
SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.0\include;%PATH%
SET PATH=C:\tools\cuda\bin;%PATH%
You can refer this link for more detailed information in this.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | TFer2 |