Lambda/ffmpeg timelapse generation - output zero bytes, can't debug ffmpeg

Question

I am attempting to use an AWS Lambda FFMPEG layer to build a timelapse of static images in an S3 bucket. To begin, I am basing my project off of the tutorial located here.

I can replicate the steps in the tutorial, so I know the FFMPEG layer is working in Lambda. I have replicated the FFMPEG commands on a standalone server, so I know they are correct.

Here is my setup: I have two S3 buckets, lambda-source-bucket and lambda-destination-bucket. The contents of lambda-source-bucket are:

1.jpg
2.jpg
3.jpg
4.jpg
5.jpg
6.jpg
7.jpg
files.txt

The files.txt contains this:

file 'https://lambda-source-bucket.s3.us-west-2.amazonaws.com/1.jpg'
file 'https://lambda-source-bucket.s3.us-west-2.amazonaws.com/2.jpg'
file 'https://lambda-source-bucket.s3.us-west-2.amazonaws.com/3.jpg'
file 'https://lambda-source-bucket.s3.us-west-2.amazonaws.com/4.jpg'
file 'https://lambda-source-bucket.s3.us-west-2.amazonaws.com/5.jpg'
file 'https://lambda-source-bucket.s3.us-west-2.amazonaws.com/6.jpg'
file 'https://lambda-source-bucket.s3.us-west-2.amazonaws.com/7.jpg'

This is my Lambda function code (in Python):

import json
import os
import subprocess
import shlex
import boto3

S3_DESTINATION_BUCKET = "lambda-destination-bucket"
SIGNED_URL_TIMEOUT = 60

def lambda_handler(event, context):

    s3_source_bucket = event['Records'][0]['s3']['bucket']['name']
    s3_source_key = event['Records'][0]['s3']['object']['key']

    s3_source_basename = os.path.splitext(os.path.basename(s3_source_key))[0]
    s3_destination_filename = "timelapse.mp4"

    s3_client = boto3.client('s3')
    s3_source_signed_url = s3_client.generate_presigned_url('get_object',
        Params={'Bucket': s3_source_bucket, 'Key': s3_source_key},
        ExpiresIn=SIGNED_URL_TIMEOUT)

    ffmpeg_cmd = "/opt/bin/ffmpeg -y -r 24 -f concat -safe 0 -protocol_whitelist file,http,tcp,https,tls -I ""https://lambda-source-bucket.s3.us-west-2.amazonaws.com/files.txt"" -c copy -s 1024x576 -vcodec libx264 -"    
command1 = shlex.split(ffmpeg_cmd)
    p1 = subprocess.run(command1, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    resp = s3_client.put_object(Body=p1.stdout, Bucket=S3_DESTINATION_BUCKET, Key=s3_destination_filename)

    return {
        'statusCode': 200,
        'body': json.dumps('Processing complete successfully')
    }

The trigger for the Lambda function is when a new files.txt file is added to lambda-source-bucket.

So far I have been able to get the trigger to fire, the function supposedly runs without errors (in Cloudwatch), and the function creates a new timelapse.mp4 in the lambda-destination-bucket. But this file is 0 bytes. I see no FFMPEG errors in the Cloudwatch console, though I am not sure I know how to configure my Lambda function code to log FFMPEG errors.

Also: if I'm going about this in a totally wrong way, I'd love to hear feedback. I'm guessing that the concat and files.txt method of looping through https:// is not the most efficient way to do this, but it's the only way I can figure this out so far.

Any help is most sincerely and humbly appreciated.

Answer 1

I am trying to do exactly the same thing, using the same tutorial.

Here is how I finally "solved" (worked around?) it:

I gave up on the concat file*. I also gave up on streaming the output directly to an s3 object. Instead I did everything on the lambda.

I downloaded each image into "/tmp" (eg /tmp/1.jpg). Then I ran the ffmpeg with the output there to (eg /tmp/timelapse.mp4). Finally I ran s3_client.upload_file() to get that mp4 into s3.

I'm not sure which part wasn't working (concat file or streaming output), and since my local lambda doesn't seem to be timing out, I'm disinclined to revert and troubleshoot (yet).

*For what it's worth, my concat file had presigned URLs, and the various docs said to add a line between each with "duration [x]" (where x is an int), and then the last line should be a repeat of just the last image.