'AWS Python Lambda - How do i run a job ONLY when a crawler is finished running?
Here is the code snippet I am working with. I need to run a job BUT ONLY after a crawler has finished running successfully:
Is it better to create a cloudwatch trigger?
many thanks
#update crawler
response = glue_client.update_crawler(
Name = 'crawler name',
Role = 'role to be used by glue to create the crawler',
DatabaseName = 'database where the crawler should create the table',
Targets =
{
'S3Targets':
[
{
'Path': event['path'] ########## get actual path
}
]
})
# invoke crawler
glue_client.start_crawler(Name = 'crawler name')
######## wait for crawler to finish running successfully #########
# invoke job
job_run = glue_client.start_job_run(
JobName = 'my_test_Job',
Arguments = {
'migration_id': event['migrationId'] }
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
