ERROR - 400 POST https://storage.googleapis.com/storage/v1/b?project=[KEY]:Invalid argument. When I try to run tasks on Airflow (Google Composer)

Question

Running my code on Composer, returns an ERROR when trying to create the bucket:

INFO - Getting connection using `google.auth.default()` since no key file is defined for hook.
INFO - Creating Bucket: tbf-processing-zone; Location: southamerica-east1; Storage Class: REGIONAL
[2022-01-14 14:05:39,048] {taskinstance.py:1152} ERROR - 400 POST https://storage.googleapis.com/storage/v1/b?project=[KEY]: Invalid argument.

My task code is:

        task_id='create_processing_bucket',
        bucket_name=PROCESSING_BUCKET_ZONE,
        project_id=PROJECT_ID,
        storage_class='REGIONAL',
        location=LOCATION,
        labels={'team':'AirFlow(Composer)'},
        gcp_conn_id="google_cloud_storage_default"
    )

Am I missing something? I will appreciate any help cause I'm trying since last night.

Answer 1

It could be for many reasons, but you can check these:

1. SDK Bug

If you are using an SDK 2.20.0, it could be a confirmed bug in the sdk.

You need to use the version 2.19.0 and it will work correctly

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
  <version>2.19.0</version>
  <scope>runtime</scope>
</dependency>

2. Airflow Connection

You need to check the configuration of the airflow.You could see this documentation about connection with airflow.

3. API Request

Validate the API request, see if the parameters you use in the URL are supported. You could see this document about API reference.

Edit

You are using this class(line 92) in your code. This parameter project_id=PROJECT_ID, which is in your code. You don’t need to add this parameter. You can see this example.

CreateBucket = GoogleCloudStorageCreateBucketOperator(
    task_id="CreateNewBucket",
    bucket_name="test-bucket",
    storage_class="MULTI_REGIONAL",
    location="EU",
    labels={"env": "dev", "team": "airflow"},
    gcp_conn_id="airflow-conn-id",
)

The parameter project_id=PROJECT_ID only is added in the template fields. You can see this code.

  template_fields: Sequence[str] = (
        'bucket_name',
        'storage_class',
        'location',
        'project_id',
        'impersonation_chain',
    )
    ui_color = '#f0eee4'