aws_boto3 adds a hash to files that I download from S3 using python

Question

Under s3 bucket, I have a list of directories, and each one contains a list of pkl files I want to download them with the same structure. This is my code to do this :

 import boto3
 s3_conn = boto3.session.Session(profile_name=profile).client('s3')
 for each in range(len(keys)):
      key = keys[each]
      prefix = file_prefix[each]
      file_name = prefix + key.split('/')[-1]
     s3_conn.download_file(bucket_name, key, os.path.join(target_dir, file_name))
 

This an example of the content of variables:

bucket_name = my_bucket
key = 'models/parent_dir/children_dir_1/model1.pkl'
target_dir = 'local_parent_dir/children_dir_1'
file_name = 'model1.pkl'

The problem that when when I try to download the directory and it's content, I found that it add a suffix to each pkl file, it's like a Hash, and this cause an error for me later. This is the error :

  s3_conn.download_file(bucket_name, key, os.path.join(target_dir, file_name))
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/boto3/s3/inject.py", line 172, in download_file
    extra_args=ExtraArgs, callback=Callback)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/boto3/s3/transfer.py", line 307, in download_file
    future.result()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/s3transfer/download.py", line 571, in _main
    fileobj.seek(offset)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/s3transfer/utils.py", line 367, in seek
    self._open_if_needed()
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/s3transfer/utils.py", line 350, in _open_if_needed
    self._fileobj = self._open_function(self._filename, self._mode)
  File "/home/ec2-user/anaconda3/envs/python3/lib/python3.6/site-packages/s3transfer/utils.py", line 261, in open
    return open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'local_parent_dir/children_dir_1/model1.pkl.e843bBf2'

All the problem cames from here : 'local_parent_dir/children_dir_1/model1.pkl.e843bBf2'.

Can any one told me the cause of this added string to the path of the file and how to avoid it.

Answer 1

you might have access issue to the destination folder, but it doesn't respond with useful information. I fixed it by granting the access to the user.