'S3ToGCSOperator : Connection error while getting file from S3 to GCS

I have been trying to source data from S3 bucket to GCS with provided operator: S3ToGCSOperator.

Below is the error I am getting:

___________________________________________________________________
environment_uri = os.environ.get(CONN_ENV_PREFIX + conn_id.upper())
AttributeError: 'Connection' object has no attribute 'upper'
___________________________________________________________________

I have tried following ways to create a connection and pass in the S3ToGCSOperator as param:

  • Creating a connection in same DAG file, using Connection()
  • Importing connection from airflow.models.connection import Connection
  • Creating the environmental variable and passing URI as value

But none of the approach has worked out, sometimes it says unable to locate credentials.

I would like to know:

  • What config shall I use for the connection to be passed in the operator
  • Where to create/declare the connection

Environment:

  • GCP
  • Cloud composer 2.1.4


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source