PermissionError: [WinError 5] Access is denied: 'foldername/ Error while S3 subfolder file download window system

Question

I am trying to download the s3 folder files into my windows system and I am getting Permission Error while executing the my python script in windows system. Any help will be highly Appreciate.

# creating folder but no data.
import boto3
import os
from pathlib import Path

s3 = boto3.resource('s3')

bucket = s3.Bucket('mybucketname')

key = 'foldername1'
objs = list(bucket.objects.filter(Prefix=key))

for obj in objs:
    # print(obj.key)

    # remove the file name from the object key
    obj_path = os.path.dirname(obj.key)

    # create nested directory structure
    Path(obj_path).mkdir(parents=True, exist_ok=True)

    # save file with full path locally
    bucket.download_file(obj.key, obj.key)

Error I am getting below:

Traceback (most recent call last):
  File "C:\MSA\EO projects\FEB 2022 WORKS\REMOTE AWZ\d6.py", line 23, in <module>
    bucket.download_file(obj.key, obj.key)
  File "C:\Program Files\Python37\lib\site-packages\boto3\s3\inject.py", line 246, in bucket_download_file
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "C:\Program Files\Python37\lib\site-packages\boto3\s3\inject.py", line 172, in download_file
    extra_args=ExtraArgs, callback=Callback)
  File "C:\Program Files\Python37\lib\site-packages\boto3\s3\transfer.py", line 307, in download_file
    future.result()
  File "C:\Program Files\Python37\lib\site-packages\s3transfer\futures.py", line 106, in result
    return self._coordinator.result()
  File "C:\Program Files\Python37\lib\site-packages\s3transfer\futures.py", line 265, in result
    raise self._exception
  File "C:\Program Files\Python37\lib\site-packages\s3transfer\tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "C:\Program Files\Python37\lib\site-packages\s3transfer\tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "C:\Program Files\Python37\lib\site-packages\s3transfer\download.py", line 601, in _main
    osutil.rename_file(fileobj.name, final_filename)
  File "C:\Program Files\Python37\lib\site-packages\s3transfer\utils.py", line 273, in rename_file
    rename_file(current_filename, new_filename)
  File "C:\Program Files\Python37\lib\site-packages\s3transfer\compat.py", line 25, in rename_file
    os.remove(new_filename)
PermissionError: [WinError 5] Access is denied: 'foldername1/'

Answer 1

When the Create Folder button is used in the Amazon S3 console, it creates a 'folder'. However, Amazon S3 does not use folders. Instead, it creates a zero-length object with the name of the folder. In this case, it created an object called folder1/.

However, when your code attempted to download this object as a file, your Operating System did not like the idea of creating a file with a name ending in a slash (/). In fact, you do not need to download this folder since the code is already using mkdir() to create the directory.

Therefore, the code can simply skip-over such objects, like this:

for obj in objs:
    if not obj.key.endswith('/'):
        # Your existing code here

Alternatively, it could skip-over zero-length objects with:

    if obj.size > 0: