SQS batching with Lambda event source mapping

Question

I am trying to do some testing with SQS and Lambda and I have set the batch size to 10 and the batch window to 10 on the event source mapping between SQS and a function.

I sent 8 messages in an 8 second window and I expected to see a single function invocation after 10 seconds with all 8 messages in the event but actually what I observed was 4 separate function invocations with differing amounts in the message bodies.

Am I misunderstanding these configuration settings, or is there something on the queue settings which is causing this?

Answer 1

I am posting this in case this catches anyone else out but I did reach out to AWS and received the following response, which I hope might prove useful to other people:

I understand that you would like to use an event source mapping to trigger Lambda functions from SQS. In testing, you have noticed some strange behavior, specifically; by setting the batch size to 10 and sending 8 messages to the queue, you expected to see a single lambda invocation containing the batch of 8 messages, but rather what you observed was 4 lambda invocations. Please feel free to correct me if I have misunderstood at any point.

To explain, in order for Lambda functions with an SQS queue configured as an event source to scale optimally, the following should be true:

- The function isn't producing any errors. 
- There are sufficient messages in the SQS queue.
- There is sufficient unreserved concurrency in the AWS Region, or the reserved concurrency for the function is at least 1,000 for standard queues or equivalent to the number of active message groups or higher for FIFO queues.

When messages are available, Lambda reads up to five batches and sends them to your function [1]. When there are fewer items in the queue, the batch will be smaller than the maximum batch size (i.e 10). Therefore, if your SQS queue has fewer than 1,000 messages, you're less likely to receive a full batch of 10 messages in your invocations.

From the above, you would only receive full batches if you have a large queue depth (i.e many messages in the queue - based on the ApproximateNumberOfMessagesVisible metric). As you were testing with a few messages and Lambda reads up to five batches when messages are available, each batch will have fewer than the batch size until there are sufficient messages in the queue to read full batches.