'Clickhouse: kafka broken messages error handling

Clickhouse version 21.12.3.32. I`m following this PR(https://github.com/ClickHouse/ClickHouse/pull/21850) to handle incorrect messages from kafka topic, but after some investigation I`ve found that if a single message contains broken data, a whole batch of received messages cannot be parsed and it can lead to data loss.

Kafka engine table:

CREATE TABLE default.kafka_engine (message String) 
ENGINE = Kafka
SETTINGS
  kafka_broker_list = 'kafka:9092',
  kafka_topic_list = 'topic',
  kafka_group_name = 'group',
  kafka_format = 'JSONAsString',
  kafka_row_delimiter = '\n',
  kafka_num_consumers = 1,
  kafka_handle_error_mode ='stream';

Example of broken message: [object Object]

First message error: JSON object must begin with '{'.: (at row 1). Other messages error: Cannot parse input: expected ']' at end of stream..

Is it possible to skip just that broken message and correctly parse other messages in a batch received from kafka topic?

Solution 1:^[1]

Changing the kafka_format to RawBLOB fixed my issue.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Allan Helfanov

'Clickhouse: kafka broken messages error handling

Solution 1:[1]

Sources

Related Questions

Solution 1:^[1]