Can Kafka Producer and Consumer be in the same Spring Boot App?

Question

I am fairly new to Kafka. I am designing and developing a Kafka solution to read Oracle tables (Kafka Connect or CDC is not an option in my organization yet) and send those large number of records to Kafka Brokers which will be eventually picked up by the Consumers and sent to the target system. Can my Kafka Producer and Consumer code lie in the same Spring Boot app? Or do I need to have my AppName-Producer and AppName-Consumer as TWO separate Spring Boot apps so that they can be scaled separately?

Let's say, I need to have 3 producers + 3 consumers (as part of one consumer group), so will that mean I need to deploy 6 instances of the same app? or 3 producer-app and 3 consumer-app to be deployed as separate applications?

Answer 1

The idea behind brokers (Kafka, ActiveMq etc...) is to share data asynchronously between diffetents application. This is part of distributed architecture patterns. That's mean diffetents applications (in the enterprise or a third party) share the same data.

Let's say, I need to have 3 producers + 3 consumers (as part of one consumer group), so will that mean I need to deploy 6 instances of the same app? or 3 producer-app and 3 consumer-app to be deployed as separate applications?

Now, to answer your questions:

• you don't need to deploy 6 instances of the same app
• you can deploy in the same application the consumer and the producer