torch: what are the differences between using net:cuda() and using cudnn.somemodules?

Question

I'm wondering what the differences are between:

1. using net:cuda() after making a neural net:

local net = nn.Sequential()
net:add(nn.SpatialConvolution(3, 8, 5, 5))
net:add(nn.View(8*20*20))
net:add(nn.Linear(8*20*20, 10))
net:add(nn.LogSoftMax())
net:cuda()

2. using cudnn.somemodules on the way completing a neural net:

local net = nn.Sequential()
net:add(cudnn.SpatialConvolution(3, 8, 5, 5))
net:add(cudnn.View(8*20*20))
net:add(cudnn.Linear(8*20*20, 10))
net:add(cudnn.LogSoftMax())

Answer 1

cuda() is a call to move tensors to the GPU for faster computation of operations.

cudnn is a library of cuda optimised modules, analogous to nn. If working on a GPU, using the cudnn analgues will be faster, but your code will not be portable to a CPU device:

NVIDIA CUDA Deep Neural Network (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. It provides highly tuned implementations of routines arising frequently in DNN applications. These release notes describe the key features, software enhancements and improvements, and known issues for the cuDNN 8.1.1 and earlier releases.

Answer 2

:cuda() is to use GPU

cudnn.xxx is to use some optimation for GPU model, so after adding it the training will be faster than only use cuda.

There is an introduction for cudnn: Accelerate Machine Learning with the cuDNN Deep Neural Network Library