'Why is Zookeeper retaining statistics of deleted znodes?
I am running Zookeeper Server v3.6.3
On my Zookeeper cluster, we currently have 5 servers configured. I have a tool configured to use Zookeeper as a lock manager for our applications. The lock manager can be very active in short bursts (1M+ znodes created/deleted within 30 minutes). I have been monitoring ZK's transactional logs, and I can see the confirmation that each node that is created is also deleted.
Problem:
Occasionally during those short bursts, Zookeeper hits its JVM Heap Space limit and goes down and must be restarted. I attempted to go to ZK's AdminServer, specifically to the 'monitor'/mntr endpoint in order to see the cluster's performance stats. What I found is that there is a breakdown of statistics for each node (5 different entries), including nodes that have already been deleted. I found documentation saying that ZK keeps a commit log of the previous 500 transactions by default, but what is being stored here is far greater than 500 now-removed nodes (more to the order of hundreds of thousands).
In addition, it has happened to me before that this 'monitor' endpoint causes the OOM error due to the vastness of data Zookeeper is keeping around.
My question is: Why and where is Zookeeper maintaining these statistics for deleted nodes, and how can I limit the amount of information being stored about them?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|