'Prometheus Compaction Issue
We are facing an issue with the Prometheus due to compaction in an OKD cluster 3.11. We have observed that Prometheus is not able to delete the .tmp files.
The Prometheus data is persisted using persistent volume as NFS.
Following are the errors:
level=error ts=2022-04-20T11:59:54.933890812Z caller=compact.go:432 component=tsdb msg="removed tmp folder after failed compaction" err="remove /prometheus/01G13DER2DGHDAQE70SXGQEQKA.tmp/chunks: directory not empty"
level=error ts=2022-04-20T11:59:54.934016435Z caller=db.go:272 component=tsdb msg="compaction failed" err="persist head block: write compaction: add series: out-of-order series added with label set \"{__name__=\\\"ALERTS\\\",alertname=\\\"ClusterMonitoringOperatorDown\\\",alertstate=\\\"pending\\\",severity=\\\"critical\\\"}\""
Pleas help me resolve this issue.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|