Apply custom function on each frame of a video with variable framerate

Question

I'm trying to apply a custom python function on every frame of a video, and create the video with modified frames as output. My input video is a mkv file, with variable framerate, and I'd like to get the same thing as output, so one frame in the input matches one in the output at the exact same time.

I tried to use this example of ffmpeg-python. However, it seems that the timestamp info are lost in the pipes. The output video has 689 frames when the input only has 300 (the durations also aren't a match, with 27s vs 11s for the input).

I also tried to first process each frame in my video and save the transformed version as PNGs. Then I "masked" the input video with the processed frames. This seems to be better because the output video has the same 11s duration than the input, but the frame count doesn't match (313 vs 300).

Code for the python-ffmpeg solution:

width = 1920
height = 1080
process1 = (
    ffmpeg
    .input(in_filename)
    .output('pipe:', format='rawvideo', pix_fmt='rgb24')
    .run_async(pipe_stdout=True)
)

process2 = (
    ffmpeg
    .input('pipe:', format='rawvideo', pix_fmt='rgb24', s='{}x{}'.format(width, height))
    .output(out_filename, pix_fmt='yuv420p')
    .overwrite_output()
    .run_async(pipe_stdin=True)
)

while True:
    in_bytes = process1.stdout.read(width * height * 3)
    if not in_bytes:
        break
    in_frame = (
        np
        .frombuffer(in_bytes, np.uint8)
        .reshape([height, width, 3])
    )

    # Just add 1 to the pixels for the example
    out_frame = in_frame + 1

    process2.stdin.write(
        out_frame
        .astype(np.uint8)
        .tobytes()
    )

process2.stdin.close()
process1.wait()
process2.wait()

Code for the overlay solution:

ffmpeg -i in.mkv -i test/%d.png -filter_complex "[0][1]overlay=0:0" -copyts out.mkv

Is there any other solution I didn't think about to perform what I'm trying to do? It doesn't seem to be that complicated but I can't find a way to do it.

Thanks for any help!

UPDATE:

Here are the logs for the input and output pipes of the python-ffmpeg solution.

Input

Input #0, matroska,webm, from 'in.mkv':
  Metadata:
    ENCODER         : Lavf59.17.100
  Duration: 00:00:11.48, start: 0.000000, bitrate: 45702 kb/s
  Stream #0:0: Video: h264 (High 4:4:4 Predictive), yuvj420p(pc, gbr/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 1k tbn (default)
    Metadata:
      ENCODER         : Lavc58.134.100 h264_nvenc
      DURATION        : 00:00:11.483000000
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> rawvideo (native))
Output #0, rawvideo, to 'pipe:':
  Metadata:
    encoder         : Lavf59.17.100
  Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24(pc, gbr/unknown/unknown, progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 2985984 kb/s, 60 fps, 60 tbn (default)
    Metadata:
      DURATION        : 00:00:11.483000000
      encoder         : Lavc59.20.100 rawvideo
frame=  689 fps=154 q=-0.0 Lsize= 4185675kB time=00:00:11.48 bitrate=2985984.1kbits/s dup=389 drop=0 speed=2.57x

Output

Input #0, rawvideo, from 'pipe:':
  Duration: N/A, start: 0.000000, bitrate: 1244160 kb/s
  Stream #0:0: Video: rawvideo (RGB[24] / 0x18424752), rgb24, 1920x1080, 1244160 kb/s, 25 tbr, 25 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
[libx264 @ 0000025afaf11140] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2 AVX512
[libx264 @ 0000025afaf11140] profile High, level 4.0, 4:2:0, 8-bit
[libx264 @ 0000025afaf11140] 264 - core 164 r3081 19856cc - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=18 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, matroska, to 'images/videos/out.mkv':
  Metadata:
    encoder         : Lavf59.17.100
  Stream #0:0: Video: h264 (H264 / 0x34363248), yuv420p(tv, progressive), 1920x1080, q=2-31, 25 fps, 1k tbn
    Metadata:
      encoder         : Lavc59.20.100 libx264
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame=  689 fps= 11 q=-0.0 Lsize= 4185675kB time=00:00:11.48 bitrate=2985984.1kbits/s dup=389 drop=0 speed=0.181x    
video:4185675kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000000%