What command shows all of the topics and offsets of partitions in Kafka

Question

I m looking for a Kafka command that shows all of the topics and offsets of partitions  If it s dynamically would be perfect  Right now I m using java code to see these information  but it s very inconvenient

User · Answer

We re using Kafka 2 11 and make use of this tool - kafka-consumer-groups     rpm -qf  bin kafka-consumer-groups confluent-kafka-2 11-1 1 1-1 noarch   For example     kafka-consumer-groups --describe --group logstash   grep -E  TOPIC filebeat  Note  This will not show information about old Zookeeper-based consumers  TOPIC            PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST             CLIENT-ID beats filebeat   0          20003914484     20003914888     404             logstash-0-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  192 168 1 1   logstash-0 beats filebeat   1          19992522286     19992522709     423             logstash-0-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  192 168 1 1   logstash-0 beats filebeat   2          19990597254     19990597637     383             logstash-0-XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  192 168 1 1   logstash-0 beats filebeat   7          19991718707     19991719268     561             logstash-0-YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY  192 168 1 2   logstash-0 beats filebeat   8          20015611981     20015612509     528             logstash-0-YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY  192 168 1 2   logstash-0 beats filebeat   5          19990536340     19990541331     4991            logstash-0-ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ  192 168 1 3   logstash-0 beats filebeat   6          19990728038     19990733086     5048            logstash-0-ZZZZZZZZ-ZZZZ-ZZZZ-ZZZZ-ZZZZZZZZZZZZ  192 168 1 3   logstash-0 beats filebeat   3          19994613945     19994616297     2352            logstash-0-AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA  192 168 1 4   logstash-0 beats filebeat   4          19990681602     19990684038     2436            logstash-0-AAAAAAAA-AAAA-AAAA-AAAA-AAAAAAAAAAAA  192 168 1 4   logstash-0   Random Tip  NOTE  We use an alias that overloads kafka-consumer-groups like so in our  etc profile d kafka sh   alias kafka-consumer-groups  KAFKA JVM PERFORMANCE OPTS   -Djava security auth login config  HOME  kafka client jaas conf    kafka-consumer-groups --bootstrap-server   KAFKA HOSTS  --command-config  etc kafka security-enabler properties

User · Answer

Kafka ships with some tools you can use to accomplish this   List topics       bin kafka-topics sh --list --zookeeper localhost 2181 test topic 1 test topic 2       List partitions and offsets       bin kafka-run-class sh kafka tools ConsumerOffsetChecker --broker-info --group test group --topic test topic --zookeeper localhost 2181 Group           Topic                  Pid Offset          logSize         Lag             Owner test group      test topic             0   698020          698021          1              test group-0 test group      test topic             1   235699          235699          0               test group-1 test group      test topic             2   117189          117189          0               test group-2   Update for 0 9  and higher  consumer APIs  If you re using the new apis  there s a new tool you can use  kafka-consumer-groups sh     bin kafka-consumer-groups sh --bootstrap-server localhost 9092 --group count errors --describe GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER count errors                   logs                           2          2908278         2908278         0               consumer-1  10 8 0 55 count errors                   logs                           3          2907501         2907501         0               consumer-1  10 8 0 43 count errors                   logs                           4          2907541         2907541         0               consumer-1  10 8 0 177 count errors                   logs                           1          2907499         2907499         0               consumer-1  10 8 0 115 count errors                   logs                           0          2907469         2907469         0               consumer-1  10 8 0 126

User · Answer

If anyone is interested  you can have the the offset information for all the consumer groups with the following command   kafka-consumer-groups --bootstrap-server localhost 9092 --all-groups --describe   The parameter --all-groups is available from Kafka 2 4 0

User · Answer

You might want to try kt  It s also quite faster than the bundled kafka-topics  This is the current most complete info description you can get out of a topic with kt  kt topic -brokers localhost 9092 -filter my topic name -partitions -leaders -replicas It also outputs as JSON  so you can pipe it to jq for further flexibility

[apache-kafka] What command shows all of the topics and offsets of partitions in Kafka?

Random Tip

Examples related to apache-kafka