'How to get record creation time from Kafka topic
We have a Kafka streams application that reads a record from one source Kafka topic does some validation, integration with some other systems via some API calls, based on integration results it builds up some other record and publishes it in a destination Kafka topic.
Both source and destination Kafka topics are created with CreateTime timestamp type. I am not sure if this timestamp type was set this way because of some business needs or just because this is the default timestamp type if you don't set it yourself.
Now we have a performance measuring tool that is capable of injecting a big load of messages into Kafka source topic as well as consuming the processing results from Kafka destination topic. When doing this it records the timestamp of each record into an embedded database for each key then evaluates the various percentiles we measure the performance for.
Because of topics timestamp type are set to CreateTime this is not working because both start time and end time have exactly the same value. Changing timestamp type to LogAppendTime for destination Kafka topic will solve our problem. However even if I am not aware about any business requirements to have CreateTime as topics timestamp types it does not mean there could not be such a requirement. Modifying your infrastructure design to satisfy your testing needs sounds to me a bad approach.
Wondering if there is another more elegant way of achieving this.
Thank you in advance for your inputs/suggestions.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
