How should I connect clickhouse to Kafka?

Question

CREATE TABLE readings_queue
(

    `readid` Int32,

    `time` DateTime,

    `temperature` Decimal(5,2)
)
ENGINE = Kafka
SETTINGS kafka_broker_list = 'serverIP:9092',
 kafka_topic_list = 'newtest',
 kafka_format = 'CSV',
 kafka_group_name = 'clickhouse_consumer_group',
 kafka_num_consumers = 3

Above is the code that I build up connection of Kafka and Clickhouse. But after I execute the code, only the table is created, no data is retrieved.

~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic newtest --from-beginning
1,"2020-05-16 23:55:44",14.2
2,"2020-05-16 23:55:45",20.1
3,"2020-05-16 23:55:51",12.9

Is there something wrong with my query in clickhouse?

When I checked the log, I found below warning.

2021.05.10 10:19:50.534441 [ 1534 ] {} <Warning> (readings_queue): Can't get assignment. It can be caused by some issue with consumer group (not enough partitions?). Will keep trying.

Answer 1

Here is an example to work with:

-- Actual table to store the data fetched from an Apache Kafka topic
CREATE TABLE data(
  a DateTime,
  b Int64,
  c Sting
) Engine=MergeTree ORDER BY (a,b);

-- Materialized View to insert any consumed data by Kafka Engine to 'data' table
CREATE MATERIALIZED VIEW data_mv TO data AS
SELECT
  a,
  b,
  c
FROM data_kafka;

-- Kafka Engine which consumes the data from 'data_topic' of Apache Kafka
CREATE TABLE data_kafka(
  a DateTime,
  b Int64,
  c Sting
) ENGINE = Kafka
SETTINGS kafka_broker_list = '10.1.2.3:9092,10.1.2.4:9092,
10.1.2.5:9092',
 kafka_topic_list = 'data_topic'
 kafka_num_consumers = 1,
 kafka_group_name = 'ch-result5',
 kafka_format = 'JSONEachRow';