Download list of specific files from AWS S3 using CLI

Question

I am trying to download only specific files from AWS. I have the list of file URLs. Using the CLI I can only download all files in a bucket using the --recursive command, but I only want to download the files in my list. Any ideas on how to do that?

Answer 1

This is possibly a duplicate of: Selective file download in AWS S3 CLI

You can do something along the lines of:

aws s3 cp s3://BUCKET/ folder --exclude "*" --include "2018-02-06*" --recursive

https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Answer 2

Since you have the s3 urls already in a file (say file.list), like -

s3://bucket/file1
s3://bucket/file2

You could download all the files to your current working directory with a simple bash script -

while read -r line;do aws s3 cp "$line" .;done < test.list

Answer 3

People, I found out a quicker way to do it: https://stackoverflow.com/a/69018735

WARNING: "Please make sure you don't have an empty line at the end of your text file".

It worked here! :-)