ffmpeg: What is the best practice to keep a live connection/socket with a camera, and save time on ffprobe

Question

Today... I used the following command: with subprocess.PIPE and subprocess.Popen in python 3:

ffmpeg -i udp://{address_of_camera} \
  -vf select='if(eq(pict_type,I),st(1,t),gt(t,ld(1)))' setpts=N/FRAME_RATE/TB \
  -f rawvideo -an -vframes {NUM_WANTED_FRAMES} pipe:`

This command helps me to capture NUM_WANTED_FRAMES frames from a live camera at a given moment.

However... it takes me about 4 seconds to read the frames, and about 2.5 seconds to open a socket between my computer and the camera's computer.

Is there a way, to have a socket/connection always open between my computer and the camera's computer, to save the 2.5 seconds?

I read something about fifo_size and overrun_fatal. I thought that maybe I can set fifo_size to be equal to NUM_WANTED_FRAMES, and overrun_fatal to True? Will this solve my problem? Or is there a different and simpler/better solution?

Should I try to record always (no -vframes flag) store the frames in a queue(With max size), and upon a wish to slice the video, read from my queue buffer? Will it work well with the keyframe?

Also... What to do when ffmpeg fails? restart the ffmpeg command?

Answer 1

FFmpeg itself is an one-n-done type of app. So, to keep the camera running, the best option is to "record always (no -vframes flag)" and handle whether to drop/record frames in Python.

So, a rough sketch of the idea:

import subprocess as sp
from threading import Thread, Event
from queue import Queue

NUM_WANTED_FRAMES = 4 # whatever it is

width = 1920
height = 1080
ncomp = 3 # rgb

framesize = width*height*ncomp # in bytes
nbytes = framesize * NUM_WANTED_FRAMES 

proc = Popen(<ffmpeg command>, stdout=sp.PIPE)
stdout = proc.stdout

buffer = Queue(NUM_WANTED_FRAMES)
req_frame = Event() # set to record, default to drop

def reader():
    while True:
        if req_frame.is_set():
            queue.put(stdout.read(nbytes))
            record_frame.clear()
        else:
            # frames not requested, drop 
            stdout.read(framesize)
    
rd_thread = threading.Thread(target=reader)
rd_thread.start()

...

# elsewhere in your program, do this when you need to get the camera data
req_frame.set()
framedata = queue.get()

....

Will it work well with the keyframe?

Yes, if your FFmpeg command has -discard nokey it'll read just keyframes.

What to do when ffmpeg fails? restart the ffmpeg command?

Have another thread to monitor the health of proc (Popen object) and if it is dead, you need to restart subprocess with the same command and overwrite with the new stdout. You probably want to protect your code with try-except blocks as well. Also, adding timeouts to queue ops would be a good idea, too.