What is a good strategy w.r.t closing/maintaining connections in kafka producers

Question

I have a queue listener application that receives messages and processes them. For each incoming message, an event is sent to a Kafka topic. The frequency of the incoming messages varies depending upon the time of the day.

What's puzzling me is this: When should I close the connection to the Kafka broker in the Producer (I'm using the java library - kafka-clients). These are some of the options I have:

• Create a new producer for each request. This seems like an overhead, especially during the peak time when the load is high

• Create a single producer for the application, and keep the connection alive by setting the config 'connection.max.idle.ms' in the broker to a high value (maybe the maximum estimated time the application remains idle?) Close the connection only when the application is shut down. This seems more acceptable than the first option, but it still has the disadvantage of holding the connection open for the idle time of the application (which could be a few hours on some days)

• Create a producer and allow it to close the connection to the broker after the max idle time (default 9 minutes). Before sending a message, check if the producer connection is alive and create a new producer if it's not. This option sounds good to me - close the connection when idle, create a new one when there's demand. But I've not come across this before, and I wonder if it perhaps has some drawbacks. And I see no method in the API (KafkaProducer) that allows me to check if the connection is alive

Has anyone faced this problem? How did you solve it?

Thanks.

Answer 1

I would suggest keeping a single instance throughout the app.

You could create a secondary heartbeat topic and send a simple message every second or minute, etc that'll keep the connection alive... With this strategy, I'd suggest reducing the retention very low of that new topic or making it compacted