how to properly use cv2.VideoWriter to write depth video and infrared video similar as cv2.imshow?

Question

I am trying to store video file from multiple sources(RGB, Depth, and infrared) from kinect sensors.

This is the image that I visualized using cv2.imshow command: enter image description here using the following code:

    cv2.imshow("ir", ir / 65535.)
    cv2.imshow("depth", depth / 4500.)
    cv2.imshow("color", color)

ir, depth both are array with size of (height, width), float32. color is a array with size of (height, width, 3), where 3 is the RGB channel and uint8 type from 0-255. Since ir and depth's value is large, we need to normalize them using the code above. And this code gave the above figures.

Now I want to store a series of image array as a video file. I use the following code:

ir_video= cv2.VideoWriter('ir.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), False)
depth_video= cv2.VideoWriter('depth.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), False)
color_video= cv2.VideoWriter('color.mp4', cv2.VideoWriter_fourcc(*'MP42'), fps, (height, width), True)
for loop: (pseudo for loop for this part, basically write every frame into the video)
    ir_video.write(ir / 65535.)
    depth_video.write(depth / 4500.)
    color_video.write(color)
ir_video.release()
depth_video.release()
color_video.release()

Color video works very well, looks very similar to the cv2.imshow command. However, ir and depth video are corrupted. All 0kb. I tried to change the fourcc code to cv2.VideoWriter_fourcc(*'mp4v'). This time the ir one saved a video that I can play. But it is very different from the cv2.imshow result. It is shown below: enter image description here

I'm wondering how I can correct save the result as I viewed in cv2.imshow command. What fourcc code should be used? Thanks a lot!

Answer 1

I worked on a similar project using other depth cameras (Orbbec, Asus Xtion) and afaik videowriter class of OpenCV does not support 16-bit depth images, that's why as suggested in the comments you should convert to 8 bit. You can take a look here for what I was using to save such a video (It's about using OpenNI2 but the main concept is there).