Is there a way to determine where messages came from in a Kafka topic?

Question

There is large amounts of data being pushed into one of our Kafka topics, is there a way to determine which producer this data is coming from?

Answer 1

You can enable quotas for the clients/users, and then monitor which clients get throttled via two quota-related JMX MBeans - bandwidth and request rate:

Metric: Bandwidth quota metrics per (user, client-id), user or client-id
MBean: kafka.server:type={Produce|Fetch},user=([-.\w]+),client-id=([-.\w]+)
What it shows:: Two attributes. throttle-time indicates the amount of time in ms the client was throttled. Ideally = 0. byte-rate indicates the data produce/consume rate of the client in bytes/sec. For (user, client-id) quotas, both user and client-id are specified. If per-client-id quota is applied to the client, user is not specified. If per-user quota is applied, client-id is not specified.

Metric: Request quota metrics per (user, client-id), user or client-id
MBean: kafka.server:type=Request,user=([-.\w]+),client-id=([-.\w]+)
What it shows: Two attributes. throttle-time indicates the amount of time in ms the client was throttled. Ideally = 0. request-time indicates the percentage of time spent in broker network and I/O threads to process requests from client group. For (user, client-id) quotas, both user and client-id are specified. If per-client-id quota is applied to the client, user is not specified. If per-user quota is applied, client-id is not specified.

Answer 2

You can use headers, and hardcode producer id in the header before producing! That's something I'm doing in Node.js using rdkafka, java also should have it!

Answer 3

1. use tcpdump on kafka broker to get packets send to 9092
2. use wireshark to filter specific kafka topic (kafka.topic.name == "xxx")
3. check the message body to get correct packet then find out source