'How to setup proxy layer on kafka broker?

I am trying to setup a proxy server (preferred in java) that can pass my connection stream to kafka broker.

Client (Consumer/Producer) <--> [PROXY SERVER] <--> Kafka Broker

My use case is consume the data from kafka, but the configuration params should be proxied to kafka broker.

e.g bootstrap.servers=xx.xx.xx.xx:9091, topic=<encrypted-topic>

Port 9091 is proxy server where I am decrypting the topic to original topic and passing the connection stream to kafka.

Job of proxy layer is to check if the encrypted topic is present in in-memory map of topics and if yes then decrypt the topic and connect to kafka broker.

javaspring-bootapache-kafkanetwork-programmingapache-kafka-security


Solution 1:[1]

You can just use HAproxy as a proxy in front of a kafka cluster.

listen kafka
    bind *:8888
    mode tcp
    balance roundrobin
    server  kafka1 127.0.0.1:8881 check
    server  kafka2 127.0.0.1:8883 check
    server  kafka3 127.0.0.1:8885 check   

listen kafka1
    bind *:8881
    mode tcp
    server  kafka1  10.10.0.1:45677 check

listen kafka2
    bind *:8883
    mode tcp
    server  kafka1 10.10.0.2:45677 check

listen kafka3
    bind *:8885
    mode tcp
    server  kafka1 10.10.0.3:45677 check

Above is a 3 node kafka cluster. Client just have to set the LBs ip:8888 as the boostrap server. Then the client will get meta data about lb_iP:8881 ,lb_iP:8883,lb_iP:8885 programmatically and will create respective bootstrap server connection paths accordingly.
By this we can do modifications to underlying real brokers ips and ports without client having to change configurations.
ref: guide with ssl setup Thanks

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1

Related Questions