'Strange behavior of AWS SQS FIFO with multiple producers per single message group
Given:
App:Spring Boot, AWS SDK v2, SqsAsyncClient
AWS:SQS FIFO with high throughput enabled, delivery delay = 0, visibility timeout = 30sContent-based deduplication enabled (although tried with messageDeduplicationId)
Consumer:
There is only one consumer with long polling(20s) and maxNumberOfMessages = 10, which just receives/deletes(sequential calls) messages
Producers:
Many producers(about 64+ threads) within one instance sending events inside a loop. All these producers use the same messageGroupId.
Issue:
I see about 10000 messages available in my queue(AWS console) but consumer does not constantly receive 10 messages as expected. Instead I see
Received messages: 0 req_duration: [20024]ms
Received messages: 10 req_duration: [22]ms
Received messages: 0 req_duration: [20030]ms
Received messages: 0 req_duration: [20022]ms
Received messages: 10 req_duration: [19]ms
Received messages: 10 req_duration: [24]ms
And this is not going to be recovered for hours or a day. Seems something is breaking there...
If I purge those 10000 messages, then produce new messages sequently then my consumer is able to consume 10000 messages within 10-20 seconds, so no any issues from consumer perspective.
Am I missing some important part?
P.S I understand there is a limit for SQS FIFO about 300TPS (send, receive, delete) but I don't see any error neither of those 3 calls.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
