Category "apache-kafka"

Scala Kafka exception: NoSuchMethodError: org.apache.avro.Schema.toString

I'm developing a kafka producer code in scala with those libs (I have to use version >6.X in kafka avro serializer to use TLS comunication): <dependency&g

Confluent connect 5.5.1 is throwing Exception: java.lang.OutOfMemoryError UncaughtExceptionHandler in thread kafka-coordinator-heartbeat-thread |

I am having a large cluster of Confluent Kafka comprising of multiple sub-clusters One for Zookeeper, another for Kafka broker with Schema Registry and KSQL str

Kafka returns "No matching PRIVATE KEY entries in PEM file" when attempting to start using PEM certificates

First of all, I've seen this thread but it's unrelated and having different issue. I have the following settings fragment in my Kafka properties file: ssl.keyst

Kafka returns "No matching PRIVATE KEY entries in PEM file" when attempting to start using PEM certificates

First of all, I've seen this thread but it's unrelated and having different issue. I have the following settings fragment in my Kafka properties file: ssl.keyst

Upgrading apache kafka to confluent kafka

We are planning to upgrade our existing apache kafka cluster to confluent kafka while upgrading do we have any data loss in the topics?? And also main reason we

How to transform a nested array into a table in ksqldb/kafka

I started learning ksqldb for Kafka and ran into a problem, I have a Product stream and the structure is the following: ID | VARCHAR(STRING) NAME

Kafka & Connect - how to fix AVRO Schema Data type

Setup Multiple independent source systems push AVRO events into a Kafka topic. A Kafka S3 sink connector reads AVRO events from this topic and writes into S3 pa

Create an open-telemetry span using trace-id and span-id in Java

I'm using open-telemetry to trace my applications and have a few microservices and Kafka broker in my distributed system. I'm using Java/spring-boot and in the

How to load multiple postgresql tables into multiple kafka topics in google cloud environment?

load multiple postgresql tables into multiple kafka topics in google cloud environment using pubsub or kafka connect.

pyspark.sql.utils.AnalysisException: Failed to find data source: kafka

I am trying to read a stream from kafka using pyspark. I am using spark version 3.0.0-preview2 and spark-streaming-kafka-0-10_2.12 Before this I just stat zoo

How to filter using ksql with array attribute type

I have a Stream on a topic with schema: --root --name: string --age: integer --accounts: Array --email I would like to select all root elements hav

How to filter using ksql with array attribute type

I have a Stream on a topic with schema: --root --name: string --age: integer --accounts: Array --email I would like to select all root elements hav

How to skip kafka history data in flink job if certain lag is encountered?

Sometimes we encounter lag in kafka consumer due to some external issues. Flink job will always consume kafka history (delayed data) with exactly-once semantics

How to skip kafka history data in flink job if certain lag is encountered?

Sometimes we encounter lag in kafka consumer due to some external issues. Flink job will always consume kafka history (delayed data) with exactly-once semantics

How to stop listening in a Spring Kafka consumer?

I use Spring for Apache Kafka. I'd like to stop listening to my topic and wait to escape OOM. How can I do it?

Can't establish SSL connection to Kafka after upgrading to python 3.7

Code I have that successfully connects to Kafka with an SSL connection in Python 3.6.7 fails when using Python 3.7.3, with error message SSL: WRONG_VERSION_NUMB

Kafka - broker partitions not in-sync after restart

We use 3 node kafka clusters running 2.7.0 with quite high number of topics and partitions. Almost all the topics have only 1 partition and replication factor o

Kafka MirorMaker2 ports for inter cluster communication

I have setup Apache MirrorMaker 3.0.0 with active-active strategy for two Kafka clusters (named DC, DR). So topic on DC is replicated by MirrorMaker2 as DC.<

Connection between kafka and spark : Failed to find data source : kafka

I am trying to do link between kafka and spark by reading data from one topic and tryy to print the content of this topic into a DataFrame, but by doing connect

Unable to connect to kafka in docker from Spring boot in host machine

I have already gone though Robin Moffat's blog, and several SO posts on the same subject, but still my configuration doesnt work. My docker-compose.yml: kafka