'Gluejob creation using gluejoboperator

Can anyone tell me how I can create a glue job using glue job operator in airflow.

job = AwsGlueJobOperator(
        task_id = 'jobCreation',
        job_name = 'jobname',
        job_desc = f"creating job creation",
        region_name = 'region',
        iam_role_name = 'role',
        num_of_dpus = 1,
        concurrent_run_limit  = 2,
        script_location = f"s3://bucketname/filename.py",
        s3_bucket =  'bucketname',
        script_args = {'connections' : 'connectionname', '--key' :'value' },
        create_job_kwargs={'GlueVersion':1}, 
        )

 

I'm getting this error by using the above code. The error is:

Invalid type for parameter MaxRetries, value: None, type: <class 'NoneType'>, valid types: <class 'int'>

So, I think I should add maxretries arg but where should I add it? And where I should pass the catalog connection names in this glue job operator? I passed it in create_kwargs as I don't know where I should pass it.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source