Prometheus show top metrics filtered by number of labels

Question

According to this: https://prometheus.io/docs/practices/instrumentation/#do-not-overuse-labels i should stick to no more than 10 labels per metric.

I'm trying to write query which shows me top 10 metrics filtered by the number of labels they have. Is it possible?

Answer 1

Prometheus provides /api/v1/status/tsdb endpoint, which exposes metric names with the highest number of labels inside seriesCountByMetricName stats. It also exposes other useful stats, which may help determining the source of high cardinality:

• labels with the highest number of unique values at labelValueCountByLabelName stats
• label=value pairs with the highest number of unique time series at seriesCountByLabelPair stats

P.S. Other Prometheus-like systems may provide additional functionality for /api/v1/status/tsdb endpoint, which can help narrowing down the source of high cardinality. For example, VictoriaMetrics provides the following additional functionality:

• Ability to specify the date for the stats via date query arg. For example, /api/v1/status/tsdb?date=2022-04-01 would return stats for April 1, 2022.
• Ability to specify the number of entries to return per each collected stats via topN query arg. For example, /api/v1/status/tsdb?topN=100 would return up to 100 entries per each stats list.
• Ability to filter out time series for stats collection via arbitrary time series selector. For example, /api/v1/status/tsdb?match[]=foobar would return stats for time series with foobar name only.

See these docs for more info.