Training YOLO on GPU can not be started

Question

I am using folowing implementation of YOLO4 on my custom data: https://github.com/taipingeric/yolo-v4-tf.keras

but after installing GPU drivers as following it does not start to train on GPU: https://www.tensorflow.org/install/gpu

$ py train.py

2022-02-22 12:37:51.088378: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2022-02-22 12:37:53.075850: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-02-22 12:37:53.077462: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library nvcuda.dll 2022-02-22 12:37:53.099088: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:21:00.0 name: Quadro P2000 computeCapability: 6.1 coreClock: 1.4805GHz coreCount: 8 deviceMemorySize: 5.00GiB deviceMemoryBandwidth: 130.53GiB/s 2022-02-22 12:37:53.099893: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2022-02-22 12:37:53.112660: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2022-02-22 12:37:53.113078: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2022-02-22 12:37:53.121451: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2022-02-22 12:37:53.124123: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2022-02-22 12:37:53.148165: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2022-02-22 12:37:53.175516: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2022-02-22 12:37:53.179125: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2022-02-22 12:37:53.179584: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2022-02-22 12:37:53.224980: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2022-02-22 12:37:53.227572: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:21:00.0 name: Quadro P2000 computeCapability: 6.1 coreClock: 1.4805GHz coreCount: 8 deviceMemorySize: 5.00GiB deviceMemoryBandwidth: 130.53GiB/s 2022-02-22 12:37:53.228411: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudart64_110.dll 2022-02-22 12:37:53.228829: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublas64_11.dll 2022-02-22 12:37:53.229262: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cublasLt64_11.dll 2022-02-22 12:37:53.229665: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cufft64_10.dll 2022-02-22 12:37:53.230053: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library curand64_10.dll 2022-02-22 12:37:53.230430: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusolver64_10.dll 2022-02-22 12:37:53.230878: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cusparse64_11.dll 2022-02-22 12:37:53.231301: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll 2022-02-22 12:37:53.231713: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2022-02-22 12:37:54.384894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2022-02-22 12:37:54.385348: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 2022-02-22 12:37:54.385591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N 2022-02-22 12:37:54.386516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3835 MB memory) -> physical GPU (device: 0, name: Quadro P2000, pci bus id: 0000:21:00.0, compute capability: 6.1) 2022-02-22 12:37:54.389180: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2022-02-22 12:37:57.190272: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2) 2022-02-22 12:38:09.307173: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library cudnn64_8.dll

version:

CUDA Toolkit 11.6.0 cuDNN 8.3.2.44 tensorflow 2.6.2