Instruct Kafka Consumer App To Start Reading From Offset

Question

If I have an application AppA that contains a Kafka consumer class, is it possible to instruct this consumer's behaviour pragmatically? For example, I may want to tell AppA over a rest API (or even via another topic) to wake up and begin consuming and processing messages from TopicB at offset or timestamp X and to stop at offset or timestamp Y. I may tell it to read the same sections of a topic repeatedly to perform different analysis of the data and I might want the consumer to sit idle when it's not performing an instruction.

Is it possible to control a consumer in this fashion? Essentially, I'm interested to know if I can read sections of topics on demand to produce processing/reports on its contents.. kind of in a similar to way to querying a relational DB via an admin console I guess.

Thanks in advance!

Answer 1

The Kafka consumer is able to consume topics at arbitrary positions.

You can use the seek() method to start consuming from a specific offset. You can also use the offsetsForTimes() method to find the offsets for a specific timestamp.

You can combine these two methods to consume specific sections of topics on demand.