AWS Lambda fan in strategy

Question

I have a python script running locally that invokes a controller lambda asynchronously which in turn invokes worker lambdas asynchronously. Each worker writes a new file to S3 bucket. The local script keeps checking the number of files in the bucket and when number of workers = number of files, assumes that all work is done. This works but I am not happy about the “reduce” part of this strategy. What is a better way of achieving this? Can SNS/SQS play a role here? How would the local script be notified after the work is complete? Thanks in advance.

Answer 1

Consider using AWS Step and feeding the job list into a Map step. This will run all operations concurrently (with a maximum concurrency of 40) and will not advance to the next phase of the Step function until all operations have completed.