'Kafka / KSQL, stuck in reducing stream/table

What I have, are two streams (from two different systems, imported via connectors). Some of the information from the different streams will be used to build combined information.

Currently, I'm working with ksqlDB but I'm having problems with the last step to reduce the information from both streams.

Both streams contains a tree structure (id/parentId), so I've used a second table for each stream to find certain information from the parents, which is then joined into a table containing all the information to do the final reduce.

The main matching column is always the same, however, one or more columns (not fixed) is also needed to do the final match. The columns might also be partial matches between them.

An example output of the table might look like this:

| id | match | matchExtra1 | matchExtra2 | matchExtra3 |
|  1 |     1 |      Extra1 |      Extra2 |      Extra3 |
|  2 |     1 |      Extra1 |      Extra4 |      Extra5 |
|  3 |     1 |      Extra6 |      Extra7 |      Extra8 |
|  4 |     1 |      Extra9 |      Extr10 |        tra8 |

In this case, id 1 and 2 should be matched and id 3 and 4 should be another match.

If this is possible within ksqlDB, that would be great. If needed to work with low-level Kafka, that's fine as long as we can achieve the end result.

Basic flow as I have it right now:

Stream/table flow



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source