How to list all available Kafka brokers in a cluster

Question

I am writing a shell script to monitor kafka brokers   I have gone through some links and found that if ZooKeeper contains a list of brokers  and if  in this list  the IP address is present  then a kafka broker is running   I want a command that I can use in my shell script to get the broker list and check whether kafka is running   Is there any curl command to get the kafka cluster status like elasticsearch

User · Answer

I did it like this     bin bash  ZK HOST  localhost  ZK PORT 2181   for i in  echo dump   nc  ZK HOST  ZK PORT   grep brokers  do     echo  i     DETAIL  zkCli -server   ZK HOST  ZK PORT  get  i 2 gt  dev null   tail -n 1      echo  DETAIL done

User · Answer

Using Confluent s REST Proxy API v3  curl -X GET -H  quot Accept  application vnd api json quot  localhost 8082 v3 clusters  where localhost 8082 is Kafka Proxy address

User · Answer

On MacOS  can try   brew tap let-us-go zkcli brew install zkcli  zkcli ls  brokers ids zkcli get  brokers ids 1

User · Answer

echo dump   nc localhost 2181   grep brokers    replace localhost with the host where zookeeper is running

User · Answer

If you are using new version of Kafka e g  5 3 3  you can use kafka-broker-api-versions  --bootstrap-server BROKER   grep 9092  You just need to pass one of the brokers

User · Answer

Here are a couple of quick functions I use when bash scripting Kafka Data Load into Demo Environments  In this example I use HDP with no security  but it is easily modified to other environments and intended to be quick and  functional rather than particularly robust   The first retrieves the address of the first ZooKeeper node from the config   ZKS1   cat  usr hdp current zookeeper-client conf zoo cfg   grep server 1       ZKS1     server 1        0-9    0-9      export ZKADDR   BASH REMATCH 1   2181 echo  using ZooKeeper Server  ZKADDR    The second retrieves the Broker IDs from ZooKeeper   echo  Fetching list of Kafka Brokers  export BROKERIDS    usr hdp current kafka-broker bin zookeeper-shell sh   ZKADDR   lt  lt  lt   ls  brokers ids    tail -1  export BROKERIDS   BROKERIDS    0-9     echo  Found Kafka Broker IDS   BROKERIDS    The third parses ZooKeeper again to retrieve the list of Kafka Brokers Host port ready for use in the command-line client   unset BROKERS for i in  BROKERIDS do DETAIL    usr hdp current kafka-broker bin zookeeper-shell sh   ZKADDR   lt  lt  lt   get  brokers ids  i       DETAIL    PLAINTEXT                  if   -z   BROKERS x     then BROKERS   BASH REMATCH 1    else  BROKERS    BROKERS    BASH REMATCH 1     fi done echo  Found Brokerlist   BROKERS

User · Answer

Alternate way using Zk-Client    If you do not prefer to pass arguments to   zookeeper-shell sh and want to see the broker details from Zookeeper CLI  you need to install standalone Zookeeper  As traditional Kafka do not comes up with Jline JAR     Once you install unzip  the standalone Zookeeper then    Run the Zookeeper CLI     zookeeper bin zkCli sh -server localhost 2181  Make sure your Broker is already running If it is successful  you can see the Zk client running as     WATCHER    WatchedEvent state SyncConnected type None path null  zk  localhost 2181 CONNECTED  0     From here you can explore the broker details using various commands        ls  brokers ids     Gives the list of active brokers    ls  brokers topics  Gives the list of topics    get  brokers ids 0  Gives more detailed information of the broker id  0

User · Answer

This command will give you the list of the active brokers between brackets     bin zookeeper-shell sh localhost 2181 ls  brokers ids

User · Answer

To use zookeeper commands with shell script try  zookeeper bin zkCli sh -server localhost 2181  lt  lt  lt   ls  brokers ids    tail -n 1  The last line usually has the response details

[apache-kafka] How to list all available Kafka brokers in a cluster?

Examples related to apache-kafka