'multi-broker kafka InvalidReplicationFactorException

I'm having some problems while getting kafka to work in a multi-broker cluster.

It's managed by Ambari 2.6.3.0 and I installed kafka broker in both hosts and I can see both Started with no alerts, but when trying to run some producer with some replication factor, it's not working.

bin/kafka-topics.sh --create --zookeeper master:2181 --replication-factor 2 --partitions 1 --topic myTopic

which result in the following error:

bigdata@master:/usr/hdp/2.6.3.0-57/kafka$ bin/kafka-topics.sh --create --zookeeper master:2181 --replication-factor 2 --partitions 1 --topic myTopic
Error while executing topic command : replication factor: 2 larger than available brokers: 1
[2017-09-19 08:45:00,430] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: replication factor: 2 larger than available brokers: 1
 (kafka.admin.TopicCommand$)

When looking in zookeeper for active brokers, I'm only getting 1 id, so second broker is not supposed to be part of the cluster.

bigdata@master:/usr/hdp/2.6.3.0-57/zookeeper$ bin/zkCli.sh 
[zk: master:2181(CONNECTED) 0] connect master:2181
[zk: master:2181(CONNECTED) 1] ls /brokers/ids
[1001]

I would thank any answer or suggestion on how to make both brokers listed together and be able to create the topic with replication factor

EDIT: add logs (pastebin not to copy so long text in here)

server.log: https://pastebin.com/rjKUxE5y

ambari:image



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source