'reading spark cluster metrics with grafana
I've a stand alone spark cluster, I'm able to get all the build in metrics for worker and driver. Following this "https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-1/" I've a Prometheus server, and set up my targets like this
- job_name: 'X_master'
metrics_path: '/metrics/master/prometheus'
static_configs:
- targets: ['X:8080']
labels:
instance_type: 'master'
spark_cluster: 'X_CLUSTER'
- job_name: 'X_spark-workers'
metrics_path: '/metrics/prometheus'
static_configs:
- targets: ['X1:8081','X2:8081']
labels:
instance_type: 'worker'
spark_cluster: 'X_CLUSTER'
- job_name: 'X_spark-driver'
metrics_path: '/metrics/prometheus'
static_configs:
- targets: ['X:4040']
labels:
instance_type: 'driver'
spark_cluster: 'X_CLUSTER'
I've grafana on same server, and tried this dashboard https://grafana.com/grafana/dashboards/7890 But how I can load data to it ? I've followed this too https://grafana.com/docs/grafana-cloud/integrations/integrations/integration-apache-spark/?pg=blog&plcmt=body-txt But how I could create grafana agent and add that yml ?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
