How to get the size of each object in S3 (all buckets) for an AWS using python boto3

Question

I am looking to find the size of each object in my S3 AWS account. Alternatively, list out objects that are more than 2 GB in Size.

I have tried listing out by bucket and I am able to get the total size:

s3 = boto3.resource('s3')
bucket = s3.Bucket('bucket-name')
size = 0

for o in bucket.objects.all():    
        size += o.size    
print ('s3 size = %.3f GB' % (size/1024/1024/1024))

I am trying to find the output as similar to the AWS CLI command which gives the object name and size.

I know S3 lists up to to 1K object (paginated) based on the request and I would have to parse it. Also, if the bucket is huge (high millions to billions) listing is going to be really rough.

Would really appreciate any inputs here.

Thanks

Answer 1

Print all objects and their size:

for o in bucket.objects.all():    
  print(o.key, o.size)   

To only print objects larger than 2GB:

for o in bucket.objects.all():  
  if o.size > 2 * 1024 * 1024 * 1024:  
    print(o.key, o.size)   

However, if you have millions of objects, I would recommend Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects (including their size).