Launch multiple ec2 instances in parallel and execute docker containers in each of them

Question

My task requires executing a python script that takes a text file and another parameter(X) as input, processes the text file, and save the output to an S3 bucket in the form of a text file.

I have an SQS queue that contains several messages with each message containing the text file name and the parameter(X) to the python file.

Now, I want to launch multiple ec2 instances to run the python script with each of the messages in SQS queue and I want to do execute these instances parallel to save time. How can I achieve this task?

In other words, how can I launch multiple ec2 instances and run the same python script (but with different inputs to the script) in parallel?

Answer 1

Your plan of putting messages into an Amazon SQS to control the processing is a good architecture. Your next choice is where to run the Compute component.

Option 1: Amazon EC2

As you suggested, you can launch multiple Amazon EC2 instances. Each instance can run your Python script and would:

• Request a message from the Amazon SQS queue
• Process the message and accompanying file (stored in Amazon S3)
• Delete the message to signal that the job was completed

Your Python script is probably single-threaded, so you could actually run multiple scripts on each instance to take advantage of the CPU power available.

You can start the script on each instance by providing a User Data script that will execute when the instance launches. This User Data shell script could download your Python script from an S3 bucket, together with any configuration files. It can then start the Python script, or even multiple copies of the script.

I also recommend that you generate a log file from each script to help diagnose any problems that may occur. The logs should be copied to an S3 bucket at regular intervals so that it survives beyond the lifetime of the instance(s).

Having the right number of EC2 instances can be an issue -- how would it know when to add more instances, or when to turn them off when there are no remaining messages? You could use Amazon EC2 Auto Scaling to control this, based on the size of the SQS queue. Running 10 Amazon EC2 instances for 1 hour costs the same as running 5 instances for 2 hours, so you might as well paralellize as much as possible.

Option 2: AWS Lambda

Your architecture also works well for using an AWS Lambda function to process the files. Think of an AWS Lambda function as a mini-EC2 instance that can start, run a script for a given input, then stop. It can also run in parallel -- the default limit is 1000 Lambda functions running in parallel, so it can be a lot more efficient than using EC2 instances. Also, you only pay for the duration while a Lambda function is running -- if the message queue is empty, there is nothing running and there is no cost involved.

Amazon SQS queues can directly trigger an AWS Lambda function. This means that you can simply put a message in the queue and the Lambda function will automatically start running.

Lambda functions automatically store their logs in Amazon CloudWatch Logs, so they are centrally available and can be examined later to debug any issues.

There are, however, some limitations with Lambda functions. They can only run for a maximum of 15 minutes. They only have 512MB of disk space available for temporary use, but you can attach an Amazon EFS filesystem to have more (permanent) storage. If your script requires complex libraries it can be a little difficult to get it working in Lambda, but there are various options (such as using Container images with the software pre-loaded).

Which to choose?

I suspect that you are more familiar with using 'servers' (Amazon EC2 instances) than AWS Lambda functions. However, your architecture appears to be ideal for using Lambda functions. I encourage you to experiment with using a Lambda function to process your files. There are lots of YouTube videos and code examples showing how to use Lambda, and you can always ask questions on StackOverflow for any specific issues you face.