'using R to upload to S3 bucket runs unto error after running for sometime

I am trying to upload some files from my local directory to S3. Each file is ~300 MB or so

library(aws.s3)

Sys.setenv(
    "AWS_ACCESS_KEY_ID" = "XXX",
    "AWS_SECRET_ACCESS_KEY" = "XXX",
    "AWS_DEFAULT_REGION" = "eu-west-2"
    )

my_buckets <- bucketlist()

file_ls <- list.files(getwd())

for(f in seq_along(file_ls)){
  
  file_ref <- file_ls[f]
  
  file_ref_path <- file.path(getwd(), file_ref)
  
  put_object(
    file = file_ref_path, 
    object = file_ref, 
    bucket = my_buckets$Bucket,
    multipart = TRUE
  )
}      

After running nicely for couple of hours, it ran into the below error which I am unable to understand.

List of 4
$ Code     : chr "InternalError"
$ Message  : chr "We encountered an internal error. Please try again."
$ RequestId: chr "xxx"
$ HostId   : chr "xxx"
- attr(*, "headers")=List of 7
..$ x-amz-request-id : chr "XXX"
..$ x-amz-id-2       : chr "XXX"
..$ content-type     : chr "application/xml"
..$ transfer-encoding: chr "chunked"
..$ date             : chr "Wed, 27 Apr 2022 00:46:39 GMT"
..$ server           : chr "AmazonS3"
..$ connection       : chr "close"
..- attr(*, "class")= chr [1:2] "insensitive" "list"

What does the error mean and how do I correct it?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source