How to list folder sizes with AWS CLI like du sh command or max depth?

Question

I can list out the buckets folders using:

aws s3 ls s3://bucket/ --recursive --human-readable --summarize

But then I also get ALL the contents of the folders also. I just want a list like:

/folder1 10GB

/folder2 6GB

This way I know where to focus and dive deeper. Is this possible because I can't find it anywhere?

Answer 1

What you are looking for is not available currently, though there is a feature request against the aws-cli project under "aws s3 ls" should have a summary-only option.

The comments also include some ideas for how you could use Bash scripting to produce what you are looking for.

Answer 2

UPDATE: fixed it. And is much simpler now and should do exactly what you want.

Trying to write a script to do this, but no promises on when or if it will ever do what you want. But you can check it here: https://github.com/thisaaronm/aws-s3-size (feel free to submit PRs too!)

Right now, it's not much any than just running the command you're running now, but I'm working through child directories in my dev branch.

And it's written pretty ugly right now.

Answer 3

A new solution in case you run across this like I did:

s3cmd ls "s3://<bucket>/<prefix>" | awk '{print $2}' | xargs -n 1 -P 20 -I{} s3cmd du -H {}