'Unable to produce to Kafka topic that is running on WSL 2 from Windows

I am running the latest Kafka on Ubuntu WSL2 successfully. I can start zookeeper, kafka server, create topics, console produce and console consume just fine from within the Ubuntu that I have running on the WSL. However, when I go into my Intellij on Windows and create a simple Java Producer it does not seem to be able to connect to the broker

Versions & Hostname

    Java version: 1.8
    Kafka Version: 2.6
    hostname (from Ubuntu): KDAAPPDEV04
    hostname (from Powershell): KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getHostName() = KDAAPPDEV04
    java.net.InetAddress.getLocalHost().getCanonicalHostName() = KDAAPPDEV04
    netstat from CMD:
        TCP    [::1]:9092             [::]:0                 LISTENING

server.properties I found this settings on another SO answer but these did not work for me.

advertised.listeners=PLAINTEXT://127.0.0.1:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

then tried (and restarted zookeeper and kafka)

advertised.listeners=PLAINTEXT://KDAAPPDEV04:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT
listeners=PLAINTEXT://0.0.0.0:9092

Producer

I run this producer with three different values: hostname, localhost and 127.0.0.1 but it never connects to the broker

    public class ProducerDemo{

    private static Logger logger = LoggerFactory.getLogger(ProducerDemo.class);

    public static void main(String[] args) throws UnknownHostException{

        System.out.println(InetAddress.getLocalHost().getHostName());
        System.out.println(InetAddress.getLocalHost().getCanonicalHostName());

        String bootstrapServers = "127.0.0.1:9092";
//        String bootstrapServers = "localhost:9092";
//        String bootstrapServers = "KDAAPPDEV04:9092";

        //create Producer properties
        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());

        //create the producer
        KafkaProducer<String,String> producer = new KafkaProducer<String, String>(properties);

        //create a producer record
        ProducerRecord<String,String> record = new ProducerRecord<String, String>("first-topic","hola mundo");

        //send data
        producer.send(record);

        //flush + close
        producer.flush();
        producer.close();
    }
}

Error

[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 2.6.0
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: 62abe01bee039651
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1601666175706
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node -1 (KDAAPPDEV04/my-ipconfig-address-here:9092) could not be established. Broker may not be available.
javaapache-kafkawindows-subsystem-for-linuxkafka-producer-apiwsl-2


Solution 1:[1]

Had this same issue. The root cause seems to be that WSL2 is broken with regards to IPv6 and localhost (See: https://github.com/microsoft/WSL/issues/4851)

The only fix I found that doesn't involve changing configs every time you reboot (per the "172.*" suggestion above) is to use the IPv6 loopback address ::1 in both the Kafka server config running in Linux and the Java client in Windows.

In server.properties I have this:

listeners=PLAINTEXT://[::1]:9092

And likewise in my Java client bootstrap server config I use

"[::1]:9092"

Solution 2:[2]

I had the exact problem you are having and I resolved it as follows:

  1. I ran the following command in my WSL2 Ubuntu shell: ip addr | grep "eth0" I made note of the ip address against the inet property, for example, 172.27.10.68
  2. In my Kafka server.properties I replaced the listeners property value as follows: listeners=PLAINTEXT://172.27.10.68:9092 I commented out the advertised.listeners property. But you can alternatively assign the ip in question to this property, and have the listeners property set to 0.0.0.0. But I assume you are using the Kafka installation for testing/learning purposes, so I would keep it simple.
  3. I made no change to the Zookeeper's default ip:port
  4. I am using the Schema Registry, so I modified the Kafka bootstrap property as follows: kafkastore.bootstrap.servers=PLAINTEXT://172.27.10.68:9092 I made no change to the default schema registry listener listeners=http://0.0.0.0:8081
  5. I used the same ip (as listed above) in my IntelliJ Kafka Producer. It then happily connected to my Kafka broker in WSL2.

More information on WSL2 networking can be found at https://docs.microsoft.com/en-us/windows/wsl/compare-versions .

The only problem with this setup is that every time you shutdown or restart your Windows machine, or close your Ubuntu terminal, the ip address for eth0 changes. And this results in redoing steps 2, 4 and 5. I am sure there is a better way, but everything I tried failed, except for this.

Solution 3:[3]

I am able to find a work around . Thanks to Goose's comments

  1. I ran the following command in my WSL2 Ubuntu shell: ip addr
  2. Then ip address against the inet property global eth0 . for example, inet 172.20.XXX.XXX/20 .... scope global eth0
  3. I replaced all localhost with this IP address in the docker-compose.yml
  4. I replaced the localhost with this IP address in springboot yml or properties file.
  5. My Kafka producer and consumer able to connect to the Kafka running in Ubunti - WSL 2 from Windows

Solution 4:[4]

Stop Kafka and Zookeeper, then

Disable IPv6 on WSL2:

sudo sysctl -w net.ipv6.conf.all.disable_ipv6=1
sudo sysctl -w net.ipv6.conf.default.disable_ipv6=1

Start Kafka, and you're good to go!

Solution 5:[5]

This is not the optimal solution, but you will be able to connect if you run your producer in Ubuntu/WSL. This means if you are using a Windows IDE, writing the code, switching to Ubuntu and using a command line compiler and running the producer. See this post Error connecting to kafka server via IDE in WSL2

Solution 6:[6]

WSL2 runs on hypervisor and you need port proxy to connect Kafka Broker running on WSL2.

Step 1 . Check you WSL2 IP using following command and copy inet value

$ ifconfig
inet 172.X.X.X

Step 2. Open cmd with Admin permsissions

   netsh interface portproxy add v4tov4 listenport=9092 listenaddress=0.0.0.0 connectport=9092 connectaddress=172.X.X.X

You should be able to connect now

Note : WSL2 IP changes everytime you restart machine

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Ed P
Solution 2 Goose
Solution 3 shijin raj
Solution 4 Stephane Maarek
Solution 5 dannylee8
Solution 6 Gautam

Related Questions