'Are there limitations with using Kafka Streams' interactive queries in production?
As the title states, have people used Kafka Streams' interactive queries for production-grade requests, or is the interactive querying more of a debugging feature?
I'm imagining that there may be issues around hot-keys if you tried to send too many QPS, which you can solve by caching, but I'm curious to know if there are more library-specific limitations.
Solution 1:[1]
It's used very frequently in production. In fact, tools like Confluent Control Center and KsqlDB heavily rely on the feature to work.
Yes, "hot keys" are a problem in any Kafka cluster, regardless of using Streams API. I'm not sure what you mean by caching, though unless you're referring to RocksDB or the in memory statestore as a cache
One main limitation is disk utilization against keyspaces with large cardinality, not "hot keys"; if the Streams app ever loses its state, it could take hours for it to rebuild. I've heard people get around this problem by writing their own state store implementations around maybe Redis or Aerospike
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | OneCricketeer |