With scala 2.11 and spark-streaming-kafka-0-8_2.11 I could do import org.apache.spark.streaming.kafka.KafkaCluster val params = Map[String, Object]( "bootstr
[2021-04-05 07:51:32,180] ERROR There was an error in one of the threads during logs loading: org.apache.kafka.common.KafkaException: Found directory /var/lib/k
I am working on a nestjs project. My project gets data from Kafka's topic and writes the data to the database (mysql). If I read hundreds of messages from Kafka
I have the following code to send data to Kafka: @Service public class KafkaSender{ @Autowired private KafkaTemplate<String, Employee> kafkaTempla
How do I set a spark job to pick up a kafka topic from a specific offset based on a timestamp ? Let's say that I need to get all data from a kafka topic startin
I am new to Kafka Streams API and I am trying to create a KTable. I have an input topic: s-order-topic, which is a json format message, as shown below. { "curr
I have a materialized view created using CREATE TABLE average_latency AS SELECT DEVICENAME, AVG(LATENCY) AS AVG_LATENCY FROM metrics WINDOW TUMBLING (SIZE 1 MIN
I am using kafka 1.0.0V In my project. From yesterday on wards. I am unable to listen Messages from command line . In the same time I am able to listen the mess
Redpanda seems easy to work with, but how would one process streams in real-time? We have a few thousand IoT devices that send us data every second. We would li
producer sends messages 1, 2, 3, 4 consumer receives messages 1, 2, 3, 4 consumer crashes/disconnects producer sends messages 5, 6, 7 consumer comes back up
I'd like to join data coming in from two Kafka topics ("left" and "right"). Matching records are to be joined using an ID, but if a "left" or a "right" record i
we are using kafka in production and I try to push the adoption and usage of KSQL in the same direction. But I already failed with one simple table-table join.
I'm writing a consumer which listens to a Kafka topic and consumes message whenever message is available. I've tested the logic/code by running Kafka locally an
I'm playing with Kafka engine on ClickHouse cluster. At the moment ClickHouse 22.1 cluster and Kafka are run in Docker. Here are configurations: https://github.
My pipeline is: Kerberized Kafka --> Logstash (hosted on a different server) --> Splunk. Can I replace the Logstash component with Kafka Connect? Could
I am reading this one: Automatic Commit The easiest way to commit offsets is to allow the consumer to do it for you. If you configure enable.auto.commit=t
I use debizium to stream postgresql data to Kafka, and use Java to subscribe Kafka topic. I receive Kafka message and get a JSON string, but
I have a couple of microservices developed using spring boot and each has its own Postgres database. These microservices exchange data with a CDC mechanism prov
I wanted to know why does co-partitioning of two Kstreams in kafka require same number of partitions for both the streams as is given in the documentation in be
I'm trying to implement a custom topology processing step implementing the Processor interface and then adding an instance of my custom processor to the topolog