'How to migrate a KSQL application?
I have a complex application flow with multiple stream-stream joins and stream-table joins. To simplify things let's say I am only using stream A and B. I tried migrating the "naive" way by dropping every stream and table and recreating them. However the problem seems to be that new data coming to stream B doesn't get joined with old data that stream A had. It only joins on new messages that have come after the recreation.
I investigated a bit more and discovered a property called 'auto.offset.reset'. As a second attempt I dropped everything again, then opened a new KSQL session and set 'auto.offset.reset' to 'earliest' and re-executed all the create statements. However the problem still seems to occur.
I have read the guides at: https://docs.confluent.io/platform/current/ksqldb/cloud-ksql-migration-guide.html and https://docs.ksqldb.io/en/latest/how-to-guides/update-a-running-persistent-query/ but they don't seem to go very in depth.
The ideal migration would be to recreate everything with new schemas and not have old messages get re-triggered or re-consumed. Also new messages should be able to join with previous messages. Any information about the best practises to migrate running KSQL applications is much appreciated.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|