'Unexpected CUDNN error (cuDNN did not return a suitable algorithm for convolution) when running open pose with C++ on Docker Container Ubuntu 20.04

I am pretty sure I have installed CUDA/CUDNN and required dependencies for running OpenPose. Here's the log that I get when executing the compiled application:

$ ./build/examples/openpose/openpose.bin -image_dir examples/media/ --write_json output/ --display 0 --render_pose 0

Starting OpenPose demo...

Configuring OpenPose...

Starting thread(s)...

Auto-detecting all available GPUs... Detected 1 GPU(s), using 1 of them starting at GPU 0.

E0426 06:40:55.717180 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.717420 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.723381 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.723603 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.768066 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.768213 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.773160 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.773306 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.780879 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.781047 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.826876 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.827036 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.838939 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.839083 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.845083 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.845235 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.849658 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.849797 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.883149 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.883289 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.889464 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.889638 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.904484 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.904656 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.931813 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.931991 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.937038 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.937199 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.943616 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.943768 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.973785 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.973924 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.977661 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.977829 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.990516 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:55.990705 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.019225 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.019363 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.022909 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.023063 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.033371 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.033520 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.036087 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.036247 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.133111 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.133284 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.143586 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.143729 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.147553 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.147673 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.246829 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.247006 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.250011 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.271068 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.277981 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.278124 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.801659 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.801812 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.816510 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.816658 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.821537 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.821761 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.824297 29992 cudnn\_conv\_layer.cpp:159\] cuDNN did not return a suitable algorithm for convolution.

E0426 06:40:56.824438 29992 cudnn\_conv\_layer.cpp:184\] cuDNN did not return a suitable algorithm for convolution.

F0426 06:40:57.426952 29992 cudnn\_conv\_layer.cpp:53\] Check failed: status == CUDNN\_STATUS\_SUCCESS (4 vs. 0)  CUDNN\_STATU

S\_INTERNAL\_ERROR

\*\*\* Check failure stack trace: \*\*\*

@     0x7fd33824b1c3  google::LogMessage::Fail()

@     0x7fd33825025b  google::LogMessage::SendToLog()

@     0x7fd33824aebf  google::LogMessage::Flush()

@     0x7fd33824b6ef  google::LogMessageFatal::\~LogMessageFatal()

@     0x7fd337e79375  caffe::CuDNNConvolutionLayer<>::LayerSetUp()

@     0x7fd337f5cbfd  caffe::Net<>::Init()

@     0x7fd337f5f000  caffe::Net<>::Net()

@     0x7fd3388fb5dc  op::NetCaffe::initializationOnThread()

@     0x7fd338915185  op::addCaffeNetOnThread()

@     0x7fd338916382  op::PoseExtractorCaffe::netInitializationOnThread()

@     0x7fd33891acc5  op::PoseExtractorNet::initializationOnThread()

@     0x7fd338912b57  op::PoseExtractor::initializationOnThread()

@     0x7fd33890e367  op::WPoseExtractor<>::initializationOnThread()

@     0x7fd3389411e7  op::Worker<>::initializationOnThreadNoException()

@     0x7fd338941338  op::SubThread<>::initializationOnThread()

@     0x7fd338942b50  op::Thread<>::initializationOnThread()

@     0x7fd3389460ec  op::Thread<>::threadFunction()

@     0x7fd33858ade4  (unknown)

@     0x7fd33828b609  start\_thread

@     0x7fd3383c9293  clone

Cuda version: 11.6

CudNN version: 8.3.2

I am running the application on Docker container Ubuntu 20.04 (nvcr.io/nvidia/tensorflow:22.01-tf2-py3) which has been configured with nvidia runtime ref: https://github.com/NVIDIA/nvidia-docker

Host OS: Windows 10 with WSL

Could anyone provide some insight about what could be causing the problem?

I have tried following solutions:

Failed to get convolution algorithm. This is probably because cuDNN failed to initialize,

https://forums.developer.nvidia.com/t/failed-to-get-convolution-algorithm-this-is-probably-because-cudnn-failed-to-initialize/67147

https://hub.docker.com/r/cwaffles/openpose