'reading spark cluster metrics with grafana

I've a stand alone spark cluster, I'm able to get all the build in metrics for worker and driver. Following this "https://dzlab.github.io/bigdata/2020/07/03/spark3-monitoring-1/" I've a Prometheus server, and set up my targets like this

  - job_name: 'X_master'
    metrics_path: '/metrics/master/prometheus'
    static_configs:
      - targets: ['X:8080']
        labels:
            instance_type: 'master'
            spark_cluster: 'X_CLUSTER'
  - job_name: 'X_spark-workers'
    metrics_path: '/metrics/prometheus'
    static_configs:
      - targets: ['X1:8081','X2:8081']
        labels:
            instance_type: 'worker'
            spark_cluster: 'X_CLUSTER'
  - job_name: 'X_spark-driver'
    metrics_path: '/metrics/prometheus'
    static_configs:
      - targets: ['X:4040']
        labels:
            instance_type: 'driver'
            spark_cluster: 'X_CLUSTER'

I've grafana on same server, and tried this dashboard https://grafana.com/grafana/dashboards/7890 But how I can load data to it ? I've followed this too https://grafana.com/docs/grafana-cloud/integrations/integrations/integration-apache-spark/?pg=blog&plcmt=body-txt But how I could create grafana agent and add that yml ?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source