'Strange behavior of AWS SQS FIFO with multiple producers per single message group

Given:

App:Spring Boot, AWS SDK v2, SqsAsyncClient

AWS:SQS FIFO with high throughput enabled, delivery delay = 0, visibility timeout = 30sContent-based deduplication enabled (although tried with messageDeduplicationId)

Consumer:

There is only one consumer with long polling(20s) and maxNumberOfMessages = 10, which just receives/deletes(sequential calls) messages

Producers:

Many producers(about 64+ threads) within one instance sending events inside a loop. All these producers use the same messageGroupId.

Issue:

I see about 10000 messages available in my queue(AWS console) but consumer does not constantly receive 10 messages as expected. Instead I see

Received messages: 0  req_duration: [20024]ms
Received messages: 10 req_duration: [22]ms
Received messages: 0  req_duration: [20030]ms
Received messages: 0  req_duration: [20022]ms
Received messages: 10 req_duration: [19]ms
Received messages: 10 req_duration: [24]ms

And this is not going to be recovered for hours or a day. Seems something is breaking there...

If I purge those 10000 messages, then produce new messages sequently then my consumer is able to consume 10000 messages within 10-20 seconds, so no any issues from consumer perspective.

Am I missing some important part?

P.S I understand there is a limit for SQS FIFO about 300TPS (send, receive, delete) but I don't see any error neither of those 3 calls.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source