'Copy s3 files to a new bucket when updated - fails with large files
I've found some code (below) which copies files to a new bucket as soon as they are updated in another. It works fine for smaller files, however ones of 5GB are not being copied, is there anything in the code which might need to be changed?
import boto3
import time, urllib.request, urllib.parse, urllib.error
import json
"""Code snippet for copying the objects from AWS source S3 bucket to target S3 bucket as soon as objects uploaded on source S3 bucket
@author: Prabhakar G
"""
print ("*"*80)
print ("Initializing..")
print ("*"*80)
s3 = boto3.client('s3')
def lambda_handler(event, context):
# TODO implement
source_bucket = event['Records'][0]['s3']['bucket']['name']
object_key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])
target_bucket = 'new-backups-keep'
copy_source = {'Bucket': source_bucket, 'Key': object_key}
print ("Source bucket : ", source_bucket)
print ("Target bucket : ", target_bucket)
print ("Log Stream name: ", context.log_stream_name)
print ("Log Group name: ", context.log_group_name)
print ("Request ID: ", context.aws_request_id)
print ("Mem. limits(MB): ", context.memory_limit_in_mb)
try:
print ("Using waiter to waiting for object to persist through s3 service")
waiter = s3.get_waiter('object_exists')
waiter.wait(Bucket=source_bucket, Key=object_key)
s3.copy(Bucket=target_bucket, Key=object_key, CopySource=copy_source)
return response['ContentType']
except Exception as err:
print ("Error -"+str(err))
return e
I've created a test case, howver when it runs it shows errors, whats causing these?
Test Case Json
{
"Records": [
{
"eventVersion": "2.0",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "1970-01-01T00:00:00.000Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "EXAMPLE"
},
"requestParameters": {
"sourceIPAddress": "127.0.0.1"
},
"responseElements": {
"x-amz-request-id": "EXAMPLE123456789",
"x-amz-id-2": "EXAMPLE123/5678abcdefghijklambdaisawesome/mnopqrstuvwxyzABCDEFGH"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "testConfigRule",
"bucket": {
"name": "removed-backups",
"ownerIdentity": {
"principalId": "EXAMPLE"
},
"arn": "arn:aws:s3:::example-bucket"
},
"object": {
"key": "HappyFace.jpg",
"size": 1024,
"eTag": "0123456789abcdef0123456789abcdef",
"sequencer": "0A1B2C3D4E5F678901"
}
}
}
]
}
Log
START RequestId: xxx123 Version: $LATEST
Source bucket : removed-backups
Target bucket : removed-backups-keep
Log Stream name: 2022/05/15/[$LATEST]removed
Log Group name: /aws/lambda/copy-files-to-keep
Request ID: removed
Mem. limits(MB): 128
Using waiter to waiting for object to persist through s3 service
Error -name 'response' is not defined
[ERROR] NameError: name 'e' is not defined
{
"errorMessage": "name 'e' is not defined",
"errorType": "NameError",
"requestId": "removed",
"stackTrace": [
" File \"/var/task/lambda_function.py\", line 35, in lambda_handler\n return e\n"
]
}
Thank you
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|