3D Rotate an image with depth map

Question

I have an RGB image of shape (h, w, 3) and a corresponding depth map of shape (h, w). Thus I know, for each pixel, its 3D coordinates.
I would like to rotate the image by some 3D rotation matrix.
I know how to apply the rotation to the input coordinates and get the coordinates in the target view, but how do I render the new view given the input image pixel values?

I tried using scipy's griddata, but this interpolation "fills" in gaps for occluded regions and overall performs interpolation, but not renering of the new view.

Is there a better way to render the new rotated view in pytorch or numpy?