'using R to upload to S3 bucket runs unto error after running for sometime
I am trying to upload some files from my local directory to S3. Each file is ~300 MB or so
library(aws.s3)
Sys.setenv(
"AWS_ACCESS_KEY_ID" = "XXX",
"AWS_SECRET_ACCESS_KEY" = "XXX",
"AWS_DEFAULT_REGION" = "eu-west-2"
)
my_buckets <- bucketlist()
file_ls <- list.files(getwd())
for(f in seq_along(file_ls)){
file_ref <- file_ls[f]
file_ref_path <- file.path(getwd(), file_ref)
put_object(
file = file_ref_path,
object = file_ref,
bucket = my_buckets$Bucket,
multipart = TRUE
)
}
After running nicely for couple of hours, it ran into the below error which I am unable to understand.
List of 4
$ Code : chr "InternalError"
$ Message : chr "We encountered an internal error. Please try again."
$ RequestId: chr "xxx"
$ HostId : chr "xxx"
- attr(*, "headers")=List of 7
..$ x-amz-request-id : chr "XXX"
..$ x-amz-id-2 : chr "XXX"
..$ content-type : chr "application/xml"
..$ transfer-encoding: chr "chunked"
..$ date : chr "Wed, 27 Apr 2022 00:46:39 GMT"
..$ server : chr "AmazonS3"
..$ connection : chr "close"
..- attr(*, "class")= chr [1:2] "insensitive" "list"
What does the error mean and how do I correct it?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
