Save cv::Mat images as raw yuv file

Question

I have a video of several cv::Mat RGB frames that I need to store as videos with lossless encoding on Jetson Xavier NX. In this process, I need to convert the RGB frames to YUV444 and then use Jetson's inbuilt lossless encoding tools to encode to h264/h265 and then save in an mp4 file format.

I can read cv::Mat frame and then convert it to the YUV444 pixel format (either manually element-wise or using OpenCV's inbuilt converter). Currently, I am writing the YUV444 frames to a binary file per frame using std::ofstream(filename, std::ios::binary). However, when I encode the yuv file to h264 or h265, I get very bad results. Is there something I am missing?

Is there a better method to save a video into a yuv file (such that I can use the encoder on the entire video).

-->> Original image in yuv444 space

-->> Image in the encoded video