zookeeper + expiring session + timeout of 8000ms exceeded

Question

We are using Kafka high level consumer / producer , and we are able to successfully consume / produce messages but the zookeeper connections keep expiring and reestablishing.

I am wondering why we get the following errors?

2022-03-04 03:04:16,000 [myid:3] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x17f4d848a4d2613, timeout of 8000ms exceeded
2022-03-04 03:04:16,000 [myid:3] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x37f4d848c941dea, timeout of 8000ms exceeded
2022-03-04 03:04:16,000 [myid:3] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x27f4d8506b32638, timeout of 8000ms exceeded
2022-03-04 03:04:16,000 [myid:3] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x37f4d848c941ded, timeout of 8000ms exceeded
2022-03-04 03:04:16,000 [myid:3] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x17f4d848a4d260f, timeout of 8000ms exceeded
2022-03-04 03:04:16,000 [myid:3] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x27f4d8506b3263c, timeout of 8000ms exceeded
2022-03-04 03:04:16,000 [myid:3] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x17f4d848a4d2616, timeout of 8000ms exceeded
2022-03-04 03:04:16,000 [myid:3] - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x27f4d8506b32633, timeout of 8000ms exceeded
2022-03-04 03:04:16,000 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@492] - Processed session termination for sessionid: 0x17f4d848a4d2613
2022-03-04 03:04:16,000 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@492] - Processed session termination for sessionid: 0x37f4d848c941dea
2022-03-04 03:04:16,000 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@492] - Processed session termination for sessionid: 0x27f4d8506b32638
2022-03-04 03:04:16,000 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@492] - Processed session termination for sessionid: 0x37f4d848c941ded
2022-03-04 03:04:16,000 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@492] - Processed session termination for sessionid: 0x17f4d848a4d260f
2022-03-04 03:04:16,000 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@492] - Processed session termination for sessionid: 0x27f4d8506b3263c
2022-03-04 03:04:16,000 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@492] - Processed session termination for sessionid: 0x17f4d848a4d2616
2022-03-04 03:04:16,000 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@492] - Processed session termination for sessionid: 0x27f4d8506b32633
2022-03-04 03:04:16,016 [myid:3] - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@643] - Got user-level KeeperException when processing sessionid:0x27f4

not sure it this post also point about our problem - https://www.ibm.com/docs/en/streams/4.3.0?topic=solutions-clients-disconnect-from-zookeeper-ensemble

note sure if ZooKeeper session timeouts are caused by "soft failures," which are most commonly a garbage collection pause. Turn on GC logging and see if a long GC occurs at the time the connection times out. Also, read about JVM tuning in Kafka.

Note we set the zookeeper.connection.timeout.ms to 600000