'S3ToGCSOperator : Connection error while getting file from S3 to GCS
I have been trying to source data from S3 bucket to GCS with provided operator: S3ToGCSOperator.
Below is the error I am getting:
___________________________________________________________________
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper())
AttributeError: 'Connection' object has no attribute 'upper'
___________________________________________________________________
I have tried following ways to create a connection and pass in the S3ToGCSOperator as param:
- Creating a connection in same DAG file, using Connection()
- Importing connection from airflow.models.connection import Connection
- Creating the environmental variable and passing URI as value
But none of the approach has worked out, sometimes it says unable to locate credentials.
I would like to know:
- What config shall I use for the connection to be passed in the operator
- Where to create/declare the connection
Environment:
- GCP
- Cloud composer 2.1.4
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
