ffmpeg overlay background color/image with multiple videos

Question

I'm new to ffmpeg and I'm stuck not sure how to continue.

I have two videos that I will like to merge into a single video with background color/image overlay.

The image bellow is what I'm trying to achieve, the rectangle is the main video, the round is another video and the red is the background color/image.

video info:
one.mp4
original size: 1280x720
position/resized: (x20, y20, w980, h:keep-aspect-ration)

two.mp4
original size: 1280x720
position/resized: (x-bottom, y-buttom-left, w:keep-aspect-ration, h200)

So far I've only been able to add a bg-color but with a problem, the audio is gone, not to say that I haven't even added the second video as part of my command.

ffmpeg -f lavfi -i color=c=white:s=1920x1080:r=24 -i video.mp4 -filter_complex "[0:v][1:v]overlay=shortest=1,format=yuv420p[out]" -map "[out]" output.mp4

Any suggestion on how to achieve it? Thanks in advance.

Answer 1

[edit 3/19: added full command for OP's need]

Try this:

ffmpeg -i one.mp4, -i two.mp4, \
  -f lavfi -i color=c=white:s=1920x1080:r=30 \
  -f lavfi -i color=c=white:s=400x400:r=1,\
              format=yuva444p,\
              geq='lum(X,Y)':'cb(X,Y)':'cr(X,Y)':\
                  a='if(lt((X-200)^2+(Y-200)^2,200^2),0,255)' \
  -filter_complex \
    [0:v]scale=980:-1[L0];\
    [1:v]crop=600:600:20:20,scale=400:-1[L1];\
    [L1][3:v]overlay=shortest=1[L2];\
    [2:v][L0]overlay=20:20:shortest=1[L3];\
    [L3][L2]overlay=W-420:H-420[vout] \
  -map [vout] -map 0:a output.mp4

• 1st color input (2:v): This serves as the background canvas. Its framerate r=30 will be the output framerate.
• 2nd color input (3:v): This is the circular mask to the cropped two.mp4. Note that I fixed geq filter so the color is properly maintained (my earlier take was made for grayscale video)
• one.mp4(0:v) gets scaled to the final size -> [L0]
• two.mp4(1:v) is first cropped to square, capturing what to be shown in the circular cutout. crop arguments are w:h:x:y where the x:y is the coordinate of the upper-left corner. The cropped frame is then scaled to the final size (400x400 in this example) -> [L1]
• The cropped and scaled two.mp4 ([L1]) is then masked by 3:v input to show through the circular cutout with the first overlay filter.
• shortest=1 option for overlay filters to end when the videos ends (not color sources which will go on forever)
• The final two overlay filters places the 2 prepared videos to desired places on the canvas. Adjust its arguments as needed to change positions
• Because SD (720p) video inputs are converted to HD (1080p) video output, you may not need to do any scaling. If so, remove the scale filters from the filtergraph. For one.mp4, the first filterchain goes, so replace [L0] label with [0:v] in the modified filtergraph.

That should do it. As we went through earlier, when you run it, make sure to remove all the spaces, newlines, and backslashes in the middle of every filtergraph expression.

[original]

An approach I implemented in the past is to manipulate the masking image with geq filter. You can try the following:

For an illustration purpose, I use a rectangle with upper-left corner at (10,10) and lower-right corner at (1500,900) and a circle with its center at (1700,950) and radius 100.

ffmpeg -i input.mp4 \
  -vf color=c=red:s=1920x1080:r=1:d=1,\
      format=rgba,\
       geq=lum='lum(X,Y)':a='if(between(X,10,1500)*between(Y,10,900)+lt((X-1700)^2+(Y-950)^2,100^2),0,255)',\
      [v:0]overlay \
      output.mp4

• color source filter only generate 1 frame (1 fps for 1 second) and overlay apply it to every frame of the input video
• color output does not have alpha channel, so you need to convert to a pix_fmt
• geq sets the alpha values of all pixels: if coordinate (X,Y) is masked (so it assigns the alpha value of 255 = opaque) or True if (X,Y) is not masked (alpha = 0, 100% transparent). See FFmpeg Expressions Documentation.
• overlay gets the input video stream v:0 and overlays the output of geq filter on it.
• Because the filtergraph is implemented with -vf output option, inpnnut audio stream will be mapped automatically

[edit single line version]

ffmpeg -i input.mp4 -vf color=c=red:s=1920x1080:r=1:d=1,format=rgba,geq=lum='lum(X,Y)':a='if(between(X,10,1500)*between(Y,10,900)+lt((X-1700)^2+(Y-950)^2,100^2),0,255)',[v:0]overlay output.mp4