'Attempting to instantiate Dataproc workflow via Cloud Scheduler results in INVALID_ARGUMENT

I have an existing, functional Dataproc workflow template, and I am attempting to create a Cloud Scheduler job to run it on a schedule. The workflow template runs without error when run via the "RUN" button on the Dataproc Workflow Template console page.

I created a Scheduler job to run this workflow template with the following command (names redacted here):

gcloud scheduler jobs create http <job-name> 
    --location=us-central1 
    --schedule="0 1 * * *" 
    --http-method=POST 
    --uri=https://dataproc.googleapis.com/v1/projects/<project-name>/regions/us-central1/workflowTemplates/<template-name>:instantiate?alt=json 
    --oauth-service-account-email=<service-account-name>@<project-name>.iam.gserviceaccount.com
    --oauth-token-scope=https://www.googleapis.com/auth/cloud-platform

The job is successfully created, but upon attempting to run it, it returns INVALID_ARGUMENT. Here is the response:

{"@type":"type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished", "jobName":"projects/<project-name>/locations/us-central1/jobs/<job-name>",
 "status":"INVALID_ARGUMENT", "targetType":"HTTP", "url":"https://dataproc.googleapis.com/v1/projects/<project-name>/regions/us-central1/workflowTemplates/<workflow-name>:instantiate?alt=json"}

I get no more descriptive response than that. There is no sign the workflow began, nor any failures listed on the Dataproc Workflow console page.

I also tried a similar Scheduler job with the same service account, scope, location, and schedule, but I cut the URL down to the base workflow location: https://dataproc.googleapis.com/v1/projects/<project-name>/regions/us-central1/workflowTemplates/<template-name> and tried --http-method=GET, and that was successful, but, of course, does not instantiate the workflow.

Is there either something I'm missing, or is there at least a better way to diagnose the issue?

google-cloud-dataprocgoogle-cloud-scheduler


Solution 1:[1]

I have now been able to successfully kick off this workflow via Scheduler once the Service Account User role has been added to the service account that is used in creating the Scheduler job. That is, the service account itself also needs to be a service account user.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 nicholasj

Related Questions