'How to control memory usage of stream created on ksql cluster
We are using KSQL Version 0.23.1. We are seeing OOM issues on ksql cluster which has persistent stateless stream. Stream is just doing the point filtering on the incoming alert using where clause. Following is sample stream
CREATE STREAM sourceStream (attrib1 VARCHAR, CUSTOMER_ID INT, userId VARCHAR) WITH (kafka_topic='source-topic-name', value_format='json');
CREATE STREAM testStream
WITH (kafka_topic='<downstream-topic-name>', partitions=<downstream-topic-partitions>
value_format='json') AS
SELECT *
FROM sourceStream
where (attrib1='Admin') and CUSTOMER_ID IN(1,2) AND userId = '052CA195-D279-4187-8A25-BFA8CC872702';
As we know from kafka consumer configuration that there are couple of configs are available to tune the control the memory usage of kafka consumer like "max.poll.records", "max.partition.fetch.bytes". I don't find this configs on confluent ksql documentation. Just wanted to check, will this config work for ksql stream or Can somebody provide pointers to any other ksql config which will help in reducing the memory usage on stream side ?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
