Category "apache-kafka"

Docker compose create kafka topics

Problem: Cannot create topics from docker-compose. I need to create kafka topics before I run a system under test. Planning to use it as a part of the pipeline,

Debezium mySql connecter not able to capture insert and updater operation

I am trying to implemnt CDC piline with Debezium mysql connecter and kafkal But Source connecter not able to pusblish event for insert and update operation in t

Configure zookeeper for kafka

I want to install kafka in my centos 6.5 machine. In kafka installation tutorial, I came to know that it needs zookeeper to run. I have already install hbase wh

Multiple postgres connections with debezium connector

When I create a kafka connect connector with the debezium connector, it results in four database connections. Three of them remain idle, while one works as the

How do I avoid data loss when using Kafka over Kubernetes and one of the nodes fail?

My application runs over a Kubernetes cluster of 3 nodes and uses Kafka to stream data. I am trying to check my system's ability to recover from node failure, s

The input line is too long when starting kafka

I'm trying to run Kafka message queue on Windows. I'm usin this tutorial - https://dzone.com/articles/running-apache-kafka-on-windows-os When i try to run it

Using Kafka as a (CQRS) Eventstore. Good idea?

Although I've come across Kafka before, I just recently realized Kafka may perhaps be used as (the basis of) a CQRS, eventstore. One of the main points that Ka

How to listen for the right ACK message from Kafka

I am doing a POC with Spring Boot & Kafka for a transactional project and I have the following doubt: Scenario: One microservices MSPUB1 receives Requests

Is it possible to add partitions to an existing topic in Kafka 0.8.2

I have a Kafka cluster running with 2 partitions. I was looking for a way to increase the partition count to 3. However, I don't want to lose existing messages

Kafka Connect failed to start

I installed kafka confluent oss 4.0 on a fresh linux centos 7 but kafka connect failed to start. Steps to reproduce : - Install Oracle JDK 8 - Copy confluen

Why Kafka so fast [closed]

If I have same hardware, to use Kafka or our current solution(ServiceMix/Camel). Is there any difference? Can Kafka handle "bigger" data than

Extracting a particular part of Kafka Topic Msg in Python

I am trying to extract the VALUE part from a bunch pf Kafka Topic Msg's in Python. I am trying to subscribe to a Kafka Topic and read the Latest Message and par

How to do type conversion when transferring data from MongoDB to kafka with debezium?

In Mongodb, the objectid is base64. I'm streaming these docs to Kafka using Debezium. How can I get ObjectId to be written as UUID in kafka? Mongo Example Doc :

Kafka Consumer is not receiving Messages on docker

I'm a begginer on kafka as well as docker, I have been doing a course and working with kafka producer and consumer but for some reason it is not working. When I

How to find the kafka version in linux

How to find the kafka version in linux? whether there is a way to find the installed kafka version other than mentioning the version while downloading it?

what is difference between partition and replica of a topic in kafka cluster

What is difference between partition and replica of a topic in kafka cluster. I mean both store the copies of messages in a topic. Then what is the real diffre

Better Event centric way to solve request-reply problem in Kafka or any streaming service

I am stuck in a problem while using Kafka in a microservice architecture . I am not able to understand how a microservice handling HTTP requests will be able to

Is this the right Kafka Consumer Config - under this Kafka setup?

My Kafka Producer is producing messages at the rate of about .. 350 mb per 30 seconds.. Kafka Setup: --> 1 Zookeeper instance --> 3 Kafka Brokers --&

Spark Streaming "ERROR JobScheduler: error in job generator"

I build a spark Streaming application to keep receiving messages from Kafka and then write them into a table HBase. This app runs pretty good for first 25 mins

Kafka MirrorMaker - from Older version to Newer Version

I want to mirroring from Kafka Source Cluster to Kafka Destination Cluster. Everything is working fine if my both Source and Target Cluster are on the same vers