tf.signal.rfft2d fft convolution centered kernel

Question

I am trying to implement an image domain convolution by multiplication of kernel and image in fourier domain. I want to speed up the operation, as my kernel is pretty large. The examples below feature the gaussian kernel, but I want to use a more complicated one which requires the 100x100 size.

I read here that the kernel should be only one quarter of the actual kernel, so the kernel center is at index 0 of the array. Does someone have an intuition for this from a theoretical perspective? Does that mean that only radial-symmetric convolution kernels can be used like this?

fft convolution code

ishape = tf.shape(x)
im_fft = tf.signal.rfft2d(x, fft_length=ishape)
kernel_fft = tf.signal.rfft2d(kernel, fft_length=ishape)
im_blurred = tf.signal.irfft2d(im_fft * kernel_fft, ishape)

centered kernel

    kernel = gaussianKernel(l=100, sigma=2)

fft shifted kernel

    kernel = gaussianKernel(l=100, sigma=2)
    kernel = np.fft.fftshift(kernel)

fft shifted and cropped to quarter

    kernel = gaussianKernel(l=100, sigma=2)
    kernel = np.fft.fftshift(kernel)[:50, :50]