ls in awscli returns "PRE". Why and how to get rid of it

Question

Using awscli in git bash, the command

aws s3 ls "s3://directory/"

returns a list of

PRE "filename"

This is inconvenient as I need to do further commands on the output and I only need the file/directory names within the given directory.

For instance, it would be nice to be able to do:

for dir in $(aws s3 ls s3://directory/) do
 aws s3 ls $dir | grep .json;
done

Any suggestions to work around this?

Answer 1

you are able to do that with something like

aws s3 ls s3://directory --recursive | awk '{print $4}' | grep .json

Answer 2

What is "PRE"?

• An S3 Prefix

You can think of prefixes as a way to organize your data in a similar way to directories. However, prefixes are not directories.

How do I get rid of "PRE" in the output?

• Use the aws s3api list-objects-v2 command which supports querying and output formatting instead of the aws s3 ls command.

aws s3api list-objects-v2 --bucket <mybucket_name> --prefix <path> --query "Contents[?contains(Key, '.json')].Key"

Note that <bucketname> here is just the name of the bucket. Do not include s3:// or any additional /.

The --query parameter is what gives you the powerful output.

For example

• --query Contents[].Key would print all of the keys.
• --query Contents[?contains(Key, '.json')].Key specifies additional criteria on the Key (the Key contains .json (e.g. the extension)) and then the final .Key tells it what attribute of the json output to return.

Answer 3

To list all folders:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep "\/$" | sed 's/\/$//'

To list all files:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep -v /$

To list all .json files:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep "\.json$"

To list all .json and all .yaml files:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep -E "(\.yaml|\.json)$"

To list all files except .json files:

aws s3 ls s3://bkt --recursive | tr -s ' ' | cut -d ' ' -f4- | grep -v "\.json$"