Concatenate multiple video files alongside delayed audio files

Question

I am currently working on a utility that is responsible for pulling audio and video files from the cloud and merging them together via FFMPEG. As I am new to FFMPEG, I am going to split the question into an FFMPEG part and a C# part just so people can answer either 1 part or the other (or both!).

FFMPEG Part

Currently, I have a working FFMPEG arg if there is only 1 video file present and it needs to be merged with multiple files.

ffmpeg -i input1.mkv -i input1.mka -i input2.mka -i input3.mka -i input4.mka -filter_complex "[1:a]adelay=0s:all=1[a1pad];[2:a]adelay=20s:all=1[a2pad];[3:a]adelay=30s:all=1[a3pad];[4:a]adelay=40s:all=1[a4pad];[a1pad][a2pad][a3pad][a4pad]amix=inputs=4:weights=1|1|1|1[aout]" -map [aout] -map 0:0 output4.mkv

The delays you see in there are determined by subtracting the start time of each file from the start time of the earliest created audio or video file. I know that if I wanted to create a horizontal stack of multiple videos, i could just do

ffmpeg -i input1.mkv -i input1.mka -i input2.mkv -i input2.mka -i input3.mka -i input4.mka
-filter_complex 
"[2:v]tpad=start_duration=120:color=black[vpad]; 
 [3:a]adelay=120000:all=1[a2pad]; 
 [4:a]adelay=180000:all=1[a3pad];
 [5:a]adelay=200000:all=1[a4pad]; 
 [0:v][vpad]hstack=inputs=2[vout]; 
 [1:a][a2pad][a3pad][a4pad]amix=inputs=4:weights=1|1|1|1[aout]" 
 -map [vout] -map [aout] 
 output.mkv

but what I want to do is both keep those delays for the audio and video files AND concatenate (not stack) those videos, how would i go about doing that?

C# Part

You see that giant arg up there? The utility is supposed to generate that based on a List of recordings. Here is the model.

List<FileModel> _records;
public class FileModel {
  public string Id { get; set; }
  public string FileType { get; set; }
  public string StartTime { get; set; }
}

The utility has to then go through that list and create the arg (as seen in the FFMPEG part) to be executed by the Xabe.FFMPEG package. The way i was thinking to approach this is to basically create 2 string builders. 1 string builder will be responsible for dealing with the inputs, the other string builder. Here is what i have so far

private async Task CombineAsync()
    {
        var minTime = _records.Min(y => Convert.ToDateTime(y.StartTime));
        var frontBuilder = new StringBuilder("-y ");
        var middleBuilder = new StringBuilder("-filter_complex \"");
        var endString = $" -map [vout] -map [aout] {_folderPath}\\CombinedOutput.mkv";

        for (var i = 0; i < _records.Count; i++)
        {
            var type = _records[i].FileType.ToLower();
            var delay = (Convert.ToDateTime(_records[i].StartTime).Subtract(minTime)).TotalSeconds;
            frontBuilder.Append($"-i {_folderPath + "\\" + _records[i].Id} ");
            var addColon = i != _records.Count - 1 ? ";" : "";
            middleBuilder.Append(type.Equals("video") ? $"[{i}:v]tpad=start_duration={delay}:color=black[v{i}pad]{addColon} " : $"[{i}:a]adelay={delay}s:all=1[a{i}pad]{addColon} ");
        }
        middleBuilder.Append("\"");
        Console.WriteLine(frontBuilder.ToString() + middleBuilder.ToString() + endString);
        // var args = frontBuilder + middleBuilder + endString;
        // try
        // {
        //     var conversionResult = await FFmpeg.Conversions.New().Start(args);
        //     Console.WriteLine(JsonConvert.SerializeObject(conversionResult));
        // }
        // catch (Exception e)
        // {
        //     Console.WriteLine(e);
        // }
    }

1. Is this the correct way to go about building the argument out?

2. How in god's name do i get something like this in there, since it relies on naming and total count for the piping and inputs=

     [0:v][vpad]hstack=inputs=2[vout]; // This part will change for video concatenation depending on what gets answered above
     [1:a][a2pad][a3pad][a4pad]amix=inputs=4:weights=1|1|1|1[aout]
   

Answer 1

In amix document
Note that this filter only supports float samples(the amerge and pan audio filters support many formats).
Maybe your files is many format, try amerge

For easy generate arguments with so much filters, try FFmpegArgs

FFmpegArg ffmpegArg = new FFmpegArg().OverWriteOutput();
List<ImageMap> imageMaps = new List<ImageMap>();
List<AudioMap> audioMaps = new List<AudioMap>();
foreach (var item in _records)
{
    if (item.IsVideo)
    {
        imageMaps.Add(ffmpegArg.AddImageInput(new ImageFileInput(item.FilePath))
            .TpadFilter().StartDuration(item.Delay).MapOut);
    }
    else
    {
        audioMaps.Add(ffmpegArg.AddAudioInput(new AudioFileInput(item.FilePath))
            .AdelayFilter().Delays(item.Delay).All(true).MapOut);
    }
}
            
var imageMap = imageMaps.HstackFilter().MapOut;
var audioMap = audioMaps.AmergeFilter().MapOut;

ffmpegArg.AddOutput(new VideoFileOutput("out.mp4", imageMap, audioMap));
var result = ffmpegArg
    .Render(c => c
        .WithFFmpegBinaryPath("path to ffmpeg.exe")
        .WithWorkingDirectory("working dir"))
    .Execute();

result.EnsureSuccess();

Or by kesh comment

FFmpegArg ffmpegArg = new FFmpegArg().OverWriteOutput();
List<ImageMap> imageMaps = new List<ImageMap>();
List<AudioMap> audioMaps = new List<AudioMap>();
foreach (var item in _records)
{
    if (item.IsVideo)
    {
        imageMaps.Add(ffmpegArg.AddImageInput(new ImageFileInput(item.FilePath))
            .TpadFilter().StartDuration(item.Delay).MapOut);
    }
    else
    {
        audioMaps.Add(ffmpegArg.AddAudioInput(new AudioFileInput(item.FilePath)));
        //audioMaps.Add(ffmpegArg.AddAudioInput(new AudioFileInput(item.FilePath).SsPosition(item.Skip)));
    }
}
var imageMap = imageMaps.HstackFilter().MapOut;
var concatFilter = audioMaps.Select(x => new ConcatGroup(x)).ConcatFilter();

ffmpegArg.AddOutput(new VideoFileOutput("out.mp4", imageMap, concatFilter.AudioMapsOut.First()));
var result = ffmpegArg
    .Render(c => c
        .WithFFmpegBinaryPath("path to ffmpeg.exe")
        .WithWorkingDirectory("working dir"))
    .Execute();

result.EnsureSuccess();