'Sense new file in AWS S3 from GCP services and move to file to GCS
Is there a way to sense new file uploaded in a aws s3 bucket (file has dynamic name and can land in any key in that specific key) and move that file to GCS.
- I tried with Airflow S3KeySensor and patterns. but feel that is it picking up every file that matches (not new one) from the s3 bucket every-time.
- Is there a way to trigger gcp cloud function with an S3Event?
amazon-s3Solution 1:[1]
This is community answer formed by users answers.
Since CF can not connect directly received a trigger from S3 amazon buckets,it is needed a Lambda function. Letting the flow on this way:
"Upload file to S3 -> Trigger AWS Lambda function -> Move file to GCS using Lambda function OR Trigger Cloud Function which would move the file to GCS"
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |