Passing parameters to Glue job from AWS Lambda

Question

We need to pass 4 parameters from AWS Lambda to AWS Glue job while triggering Glue job.

response = client.start_job_run(JobName = 'my_test_Job',
         Arguments = {
           '--yr_partition_val':   2017,
           '--mon_partition_val':  05,
           '--date_partition_val':  25,
           '--hour_partition_val': 07 } )

Glue need to catch these 4 parameters to proceed further in pyspark glue code.

I have tried using below in glue to catch parameters:

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv,
                      ['JOB_NAME',
                       'yr_partition_val',
                       'mon_partition_val',
                       'date_partition_val',
                       'hour_partition_val'])

but got the error as:

self.error(_('argument %s is required') % name)
awsglue.utils.GlueArgumentError: argument --JobName is required

Can someone help it out?

Answer 1

glueJobName = 'job_01'
for record in event['Records']:
    print(record['s3']['object']['key'])
    file_name = unquote_plus(record['s3']['object']['key'])
    client = boto3.client('glue')
    print("Job is going to start")
    arguments = {
        '--source': 'abc',
        '--bucket_name': 'test',
        '--folder_name': 'abc-2',
        '--file_name': file_name,
    }
    response = client.start_job_run(JobName = 'job_01', Arguments=arguments)
return {
    'statusCode': 200,
    'body': json.dumps('Glue Trigger Lambda!' )
}