'Gluejob creation using gluejoboperator
Can anyone tell me how I can create a glue job using glue job operator in airflow.
job = AwsGlueJobOperator(
task_id = 'jobCreation',
job_name = 'jobname',
job_desc = f"creating job creation",
region_name = 'region',
iam_role_name = 'role',
num_of_dpus = 1,
concurrent_run_limit = 2,
script_location = f"s3://bucketname/filename.py",
s3_bucket = 'bucketname',
script_args = {'connections' : 'connectionname', '--key' :'value' },
create_job_kwargs={'GlueVersion':1},
)
I'm getting this error by using the above code. The error is:
Invalid type for parameter MaxRetries, value: None, type: <class 'NoneType'>, valid types: <class 'int'>
So, I think I should add maxretries arg but where should I add it? And where I should pass the catalog connection names in this glue job operator? I passed it in create_kwargs as I don't know where I should pass it.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
