'AWS s3 cp with random date
i´m a newbie. My goal is to parametrize a date path in aws s3 cp command inside a pyspark script. The files are uploaded once a month in a undefined day, say process_date = DDMMYYYY where MM is current month and YYYY is current year
This is the command:
os.system(f'aws s3 cp {root_folder}{date_of_file_upload}/ {destination_folder} --recursive')
and these are the variable i´m using:
process_monthyear = process_date[2:4] + process_date[4:]
root_folder = 's3://company-id-files/'
destination_folder = '/tmp/company-id-files/'
date_of_file_upload = f'*{lit(process_monthyear)}' #wildcard * won´t work here
Files in S3 folders in DDMMYYYY format
Considering image above, if i manually input date_of_file_upload = ´11092021´ everything works nicely. The problem is.. since i´m unable to control future upload dates (what day of the month the files will be uploaded), is there a way to parametrize this date_of_file_upload variable?
Thanks!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
