AWS SQS groups in FIFO queues

Question

So from the SQS documentation, consumer can be scaled horizontally, if group -ids are added to messages. I assume that these group-ids are dynamic.

If a consumer is initially handling messages of group-id A and B, and a big inflow of A comes in, would messages of group B be automatically routed to another consumer if no messages of type B are currently in-flight ? How is the consumer client routing determined, is it dynamic, or a fixed hash ? eg: if new group-ids are added is it ensured that the load is distributed evenly across consumers ?

Is there a way to get group-wise counts of the queue size ?

Answer 1

It is your consumers which would be requesting messages from SQS.

If all messages are grouped, and a consumer takes a message from group A, all other messages from group A would be invisible to the other consumers until the message is acknowledged (deleted) or reaches visibility timeout. Therefore, on any other consumer's request, they will be given messages from other visible (unblocked) groups - in this case group B (as long as there are less than 20k messages for group A in the queue, as per documentation).

NB: In a FIFO grouped multi-consumer setup like this, I would suggest against prefetching more than 1 message per consumer. Let's say you have prefetch 10, and a consumer requests and gets 10 messages (from 10 different groups as explained above). My understanding is that this will block all 10 groups in this consumer's own prefetched "queue", and undermine the multi-consumer setup.

NB2: If you want to reject a message (eg. send it to deadletter), make sure to call the ChangeMessageVisibility to set it to 0, so it is rejected immediately. If not, SQS will be waiting until the queue visibility timeout is reached, blocking that group for the whole time it is waiting.