'Throughput For Each Worker Goes Down When Decreasing Parallelism
I have a simple stateless Flink application running on Kubernetes that reads from a Kafka topic and prints out content. The input TPS is consistently high (> 3K) and the number of partitions on that topic is 180. When running with higher parallelism (for example 3), I notice that each worker can process up to 1K records/sec. However, when I have a lower degree of parallelism (2 and 1), the throughput for each worker significantly goes down, sometimes even to 0 records/sec. What could be the reason for such behavior? And would it be possible to mitigate? I am running on Flink 1.12, with unaligned checkpointing enabled. I also checked if there was back pressure on any of the operators, but all show OK.
Thank you!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
