'Airflow DAG Task to Archive files creates recursive folders
I have the below Google GCS to GCS file move. In Google GCS (Storage) I have a folder in my bucket (we'll call "bucket1"). In bucket1, there's an Archive Folder, and a series of json files.
I'm trying to put the json files into the Archive folder. My problem is that it's also making a copy of the Archive folder (plus contents inside). This is creating a recursive Archive folder (i.e. Archive\Archive\Archive....). Here's the task:
archive_files = GoogleCloudStorageToGoogleCloudStorageOperator(
task_id='Archive_Files',
source_bucket='my_data',
source_object='*.json',
destination_bucket='my_data',
destination_object='Archive/',
move_object=True,
google_cloud_storage_conn_id='connection1',
dag=dag
)
How do I tell Airflow, I only want the files in that folder, whithout creating a recursive "Archive" folder?
Thanks!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
