How can i implement a Queue in express App

Question

I'm working with an express app which is deployed in a EC2 container. This app gets the request from anAWSLambda with some data to handle a Web Scrapping service (is deployed in EC2 because deploying it in AWSLambda is difficult). The problem is that I need to implement a queue service in the express app to avoid opening more than X quantity of browsers at the same time depending the instance size. How can I implement a queue service to await for a web scrapper request to be terminated before launching another one? The time is not a problem because is a scheduled task that executes in early morning.

Answer 1

A simple in memory queue would not be enough, as you would not want request to be lost incase there is a crash.

If you are ok with app crash or think there there is very low probability then node modules like below could be handy. https://github.com/mcollina/fastq

If reliability is important then amazon SQS should be good to go. https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/sqs-examples-send-receive-messages.html

Queue the work on request handler. Have a simple timeout base handler which can listen to queue and perform task.