Dynamodb stream lambda, how to infinite block a partition-key stream

Question

I’m using Dynamodb stream lambda to maintain a kind of business sequence logic.

In case of failure, I can only block processing on the affected records for up to one day. Once they are expired, lambda will continue to process the next non-expired ones. Thus my sequence logic will be corrupted.

My understanding is Dynamodb stream lambda ensures in-order processing at the table partition-key level.

It would be great if I could implement a partial infinite blocking for a specific sub-stream (aka a table partition-key stream).

I’m thinking to use:

• MaximumRecordAgeInSeconds to limit the retry logic (up to a few hours to allow hotfixes)
• an SQS FIFO queue as a destination for discarded records
• a second Dynamodb table for lock purpose (pay per request billing mode)

The lambda handler has to be aware of MaximumRecordAgeInSeconds and set the lock just before discarding the batch of records in case of error.

The lock value will force the following records of the concerned partition-key sub-stream to fail.

Finally, an SQS consumer will deal with the blocking issue, process the discarded records from the queue, then unlock the sub-stream.

The locking logic looks hacky and still has some pitfalls

Can I do better?