'AWS s3 cp with random date

i´m a newbie. My goal is to parametrize a date path in aws s3 cp command inside a pyspark script. The files are uploaded once a month in a undefined day, say process_date = DDMMYYYY where MM is current month and YYYY is current year

This is the command:

os.system(f'aws s3 cp {root_folder}{date_of_file_upload}/ {destination_folder} --recursive')

and these are the variable i´m using:

process_monthyear = process_date[2:4] + process_date[4:]
root_folder = 's3://company-id-files/'
destination_folder = '/tmp/company-id-files/'
date_of_file_upload = f'*{lit(process_monthyear)}' #wildcard * won´t work here

Files in S3 folders in DDMMYYYY format

Considering image above, if i manually input date_of_file_upload = ´11092021´ everything works nicely. The problem is.. since i´m unable to control future upload dates (what day of the month the files will be uploaded), is there a way to parametrize this date_of_file_upload variable?

Thanks!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source