Append a file in S3 using Lambda [closed]

Question

How can I create and append a file in S3 using a lambda function? Also I want to limit the file size to 5 MB, is it possible?

Say, my lambda function searches for file, I want to keep a backup of the file searched. So I want to append the file whenever something is searched. If the file size increases above 5 MB I want to create a new file, which can be named according to the date and then start appending into it. How can I do that? I am using python/boto3. –

Answer 1

Well you can not append any file in S3 as it is an "object" based storage. If you have to modify a file, you have to read it, append it, and save it again with same filename to update it on S3. There are tons of blogs on how to save file in s3 from lambda. It depends on your technology and framework.

One of such blogs: https://foobar123.com/serverless-save-a-file-in-s3-using-aws-lamdba-d3f087880dd2

Answer 2

You can use the AWS SDK in Lambda functions to create a file in S3. For example, for Python, there is boto3.

However, I don't think there is a way to edit an S3 object directly.

You can download the file, append, and re-upload. Although this is not ideal because you keep downloading and re-uploading. Depending on the use case, you may want to consider other options that do not involve appending the file. For example, just create new objects with the additional lines.

Answer 3

I don't recommand use lambda to upload file in S3 ( payload limit, time of treatment...)

I suggest you generate a pre-signed url from the lambda to upload the file directly to S3, the benefit of this is you upload the file directly from your local to S3 ( you can do a multipart upload, Transfert acceleration...).

The majority of projects that use lambdas for upload file, have faced this limitation.

Think of lambda like a rapid service to give you the url to upload, and you got another lambda that perform treatment once the file has been uploaded ( trigger from object put notification...)