'Feed Prometheus from a Kafka bridge
We try to feed metrics from worldwide distributed devices into Prometheus. To decouple the devices from the services (to avoid too many direct communications), there is a message broker and a Kafka bridge.
+----------------------------------+
| Device 1 |
| +---------------+ |
| | Application I |--+ |
| +---------------+ \ |
| | |
| +----------------+ | +------+ | +------+ +----------+ +--------------+
| | Application II |---+->| MQTT |-|---metrics--->| MQTT |---->| Kafka |------>| Prometheus |
| +----------------+ | +------+ | +------+ +----------+ +--------------+
| | |
| +-----------------+ / |
| | Application III |-+ |
| +-----------------+ |
+----------------------------------+
We struggle to figure out how to feed Prometheus from the Kafka bridge more or less directly. We could - of course - instantiate translation services per Application (Service) on each device, but this feels being overkill.
We didn't find any suggestion on https://prometheus.io/docs/instrumenting/exporters/ - but I think we ask the wrong questions.
Our first application is a collectd to measure some common system metrics. This is a reasonable data source since everyone knows what should be seen in Prometheus then.
The communication
+------+ +------+ +----------+
| MQTT |----Internet--->| MQTT |---->| Kafka |
+------+ +------+ +----------+
cannot reasonable opened up for particular messages, since each message shall receive Kafka.
Solution 1:[1]
There is only a remote write integration for Kafka from Prometheus.
If you intend to collect metrics from external resources, you wouldn't write to Kafka at all, rather add some exporter (ex. collectd exporter) to the applications themselves, then setup scrape targets against them since Prometheus is pull-based.
If you really wanted to use Kafka, you would need to write a consumer first, then use PushGateway to push data; there is nothing provided by Prometheus project for this, that I am aware of. Or you can basically rewrite (or pay Confluent for) that Kafka Connect project mentioned here.
One alternative, install a StatsD server as a relay - https://github.com/prometheus/statsd_exporter#relaying-from-statsd
Push to that, rather than Kafka. (Or again, consume from Kafka, and push to the statsd server)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |