Add GCP credentials to airflow via command line

Question

Airflow allows us to add connection information via command-line airflow connections. This can help with automated deployment of airflow installations via ansible or other dev-ops tools.

It is unclear how connections to google cloud platform (service accounts) can be added to ariflow via command line.

Answer 1

Pre airflow 1.9 the following example outlines how to use a DAG to add connection information: https://gist.github.com/yu-iskw/42f9f0aa6f2ff0a2a375d43881e13b49

def add_gcp_connection(ds, **kwargs):
    """"Add a airflow connection for GCP"""
    new_conn = Connection(
        conn_id=<CONNECTION_ID>,
        conn_type='google_cloud_platform',
    )
    scopes = ['https://www.googleapis.com/auth/cloud-platform']
    conn_extra = {
        "extra__google_cloud_platform__scope": ",".join(scopes),
        "extra__google_cloud_platform__project":
            "<GCP_PROJECT_NAME>",
        "extra__google_cloud_platform__key_path":
            "<GCP_CREDENTIALS_ABSOLUTE_PATH.json>"
    }
    conn_extra_json = json.dumps(conn_extra)
    new_conn.set_extra(conn_extra_json)

    session = settings.Session()
    session.add(new_conn)
    session.commit()

From airflow 1.9 forward one can:

airflow connections -a \
  --conn_id=<CONNECTION_ID> \
  --conn_type=google_cloud_platform \
  --conn_extra='{ "extra__google_cloud_platform__key_path":" '`
        `'<GCP_CREDENTIALS_ABSOLUTE_PATH.json>", '`
    `'"extra__google_cloud_platform__project": '`
        `'"<GCP_PROJECT_NAME>", '`
    `'"extra__google_cloud_platform__scope":  '`
        `'"https://www.googleapis.com/auth/cloud-platform"}'

Answer 2

Since this was posted, documentation has been added for Google Cloud connections: https://airflow.apache.org/docs/apache-airflow-providers-google/stable/connections/gcp.html

As an alternative to using --conn_extra, which can can make it challenging to substitute values into the json string, you can use the conn_uri. For example:

# AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT='google-cloud-platform://'
# GCP_PROJECT='<your-gcp-project>'
airflow connections \
  --add \
  --conn_id '<CONNECTION_ID>' \
  --conn_uri "${AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT}?extra__google_cloud_platform__project=${GCP_PROJECT}"

Where you can add as many additional uri params as needed, separated by &s.

Answer 3

Another more readable example which works in Airflow 2.2.5.
Here the JSON file test.json:

{"test2_gcp_connection": {
"conn_type": "google_cloud_platform",
"description": null,
"host": null,
"login": null,
"password": null,
"schema": null,
"port": null,
"extra": "{ \"extra__google_cloud_platform__key_path\": \"/path/to/key.json\", \"extra__google_cloud_platform__project\": \"project_name\", \"extra__google_cloud_platform__scope\": \"https://www.googleapis.com/auth/cloud-platform\"}"}}

You can then use the command:

airflow connections import test.json

And your GCP connection will be created.

In a similar way, you can export an already created airflow connection in a file named test.json using the command:

airflow connections export test.json

Another way to write it in a similar but more simplified way than the first answer:

airflow connections add 'test_gcp_connection' --conn-type=google_cloud_platform --conn-extra='{ "extra__google_cloud_platform__key_path": "path/to/key.json", "extra__google_cloud_platform__project": "projectname", "extra__google_cloud_platform__scope": "https://www.googleapis.com/auth/cloud-platform"}'