'Spark Prometheus metrics name contain random generated data?
Metric names that Spark 3.2.0 generates are as follows:
metrics_spark_a929f03bcd084dd08ffe5a3c9ba71018_driver_DAGScheduler_stage_waitingStages_Number{type="gauges"} 0
metrics_spark_a929f03bcd084dd08ffe5a3c9ba71018_driver_DAGScheduler_messageProcessingTime_50thPercentile{type="timers"} 24749.0
I suspect that the part a929f03bcd084dd08ffe5a3c9ba71018 is random generated. How can one write proper Grafana queries to match these metrics? Is it possible to configure Spark not to add these hashes into the metric names? Whare is the purpose for these random hashes in the metrics naming?
(Checked Spark's example spark.metrics.conf for 3.2.0, but found no option to configure this.)
Solution 1:[1]
In Spark properties, you can specify following:
"spark.metrics.namespace": "myapp"
So you will obtain:
metrics_myapp_driver_BlockManager_memory_maxOffHeapMem_MB_Number{type="gauges"} 0
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Gregoire |