Category "google-cloud-composer"

configuring Airflow ssh connection in google cloud composer

I'm trying to configure a SSH connection from Airflow UI on google cloud composer environment to an on premise posgresql server Where I should store my private

google cloud composer library installation failure

while attempting to update the installed pypi packages on my composer env, I'm getting the following error - UPDATE operation on this environment failed 7 minu

S3ToGCSOperator : Connection error while getting file from S3 to GCS

I have been trying to source data from S3 bucket to GCS with provided operator: S3ToGCSOperator. Below is the error I am getting: ______________________________

GCP-Cloud Composer: Secret Manager access variable.json

I try to configure Secret Manager for my Composer (ver 1.16, airflow 1.10) but I have a weird situation like below. In my Composer, I've used a variable.json fi

Airflow DAG fails when PythonOperator with error "Negsignal.SIGKILL"

I am running Airflowv1.10.15 on Cloud Composer v1.16.16. My DAG looks like this : from datetime import datetime, timedelta # imports from airflow import DAG fr

KubernetesPodOperator is not able to access the Airflow Connections stored in Google Secrets Manager

I am using the Composer version 2.0.0 Airflow 2.1.4 and I have created a KubernetesPodOperator that is trying to access the Airflow connection stored in the Goo

BigQueryInsertJobOperator dryRun is returning success instead of failure on composer (airflow)

When using BigQueryInsertJobOperator and setting the configuration to perform a dry run on a faulty .sql file/ a hardcoded query, the task succeeds even though

loop over airflow variables issue question

I am having hard time looping over an airflow variable in my script so I have a requirement to list all files prefixed by string in a bucket. next loop throug