'How to get a reference to previous event for a particular grouping using ksqlDB or kafka streams?

I have a stream of events called transactions. Over time, many transactions are committed by a single address and there are many addresses.

For every transaction a particular address make, I need a way to see the previous transaction the address made. Essentially, I need to create a singularly linked list of the transactions for a particular address.

Using the aggregate functions available in ksqlDB at the moment, this doesn't seem possible (however, I'd love to be proven wrong). Also, without multi-column joins, it becomes prohibitively expensive to join the incoming transaction stream with a table of previous transactions as tx_with_prev grows exponentially with the number of transactions:

with tx_with_prev as (
  select  tx_stream_id
      ,   tx_id
  from tx_stream 
  left join tx 
  on tx_stream.address = tx.address
)

select  tx_stream_id
    ,   max(tx_id) as tx_id
from tx_with_prev
where tx_stream_id > tx_id
group by tx_stream_id

Wondering if there's any other way to do this in ksqlDB or if there is a way to do it with kafka streams which I am completely unfamiliar with.

Any input is appreciated!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source