changing kafka retention period during runtime

Question

With Kafka 0 8 1 1  how do I change the log retention time while it s running  The documentation says the property is log retention hours  but trying to change it using kafka-topics sh returns this error    bin kafka-topics sh --zookeeper zk yoursite com --alter --topic as-access --config topic log retention hours 24 Error while executing topic command requirement failed  Unknown configuration  topic log retention hours   java lang IllegalArgumentException  requirement failed  Unknown configuration  topic log retention hours       at scala Predef  require Predef scala 145      at kafka log LogConfig  anonfun validateNames 1 apply LogConfig scala 138      at kafka log LogConfig  anonfun validateNames 1 apply LogConfig scala 137      at scala collection Iterator class foreach Iterator scala 631      at scala collection JavaConversions JEnumerationWrapper foreach JavaConversions scala 479      at kafka log LogConfig  validateNames LogConfig scala 137      at kafka log LogConfig  validate LogConfig scala 145      at kafka admin TopicCommand  parseTopicConfigsToBeAdded TopicCommand scala 171      at kafka admin TopicCommand  anonfun alterTopic 1 apply TopicCommand scala 95      at kafka admin TopicCommand  anonfun alterTopic 1 apply TopicCommand scala 93      at scala collection mutable ResizableArray class foreach ResizableArray scala 57      at scala collection mutable ArrayBuffer foreach ArrayBuffer scala 43      at kafka admin TopicCommand  alterTopic TopicCommand scala 93      at kafka admin TopicCommand  main TopicCommand scala 52      at kafka admin TopicCommand main TopicCommand scala

User · Accepted Answer

log retention hours is a property of a broker which is used as a default value when a topic is created  When you change configurations of currently running topic using kafka-topics sh  you should specify a topic-level property  A topic-level property for log retention time is retention ms  From Topic-level configuration in Kafka 0 8 1 documentation   Property  retention ms Default  7 days Server Default Property  log retention minutes Description  This configuration controls the maximum time we will retain a log before we will discard old log segments to free up space if we are using the  quot delete quot  retention policy  This represents an SLA on how soon consumers must read their data   So the correct command depends on the version  Up to 0 8 2  although docs still show its use up to 0 10 1  use kafka-topics sh --alter and after 0 10 2  or perhaps from  0 9 0 going forward  use kafka-configs sh --alter   bin kafka-topics sh --zookeeper zk yoursite com --alter --topic as-access --config retention ms 86400000    You can check whether the configuration is properly applied with the following command    bin kafka-topics sh --describe --zookeeper zk yoursite com --topic as-access  Then you will see something like below  Topic as-access  PartitionCount 3  ReplicationFactor 3  Configs retention ms 86400000

User · Answer

The following is the right way to alter topic config as of Kafka 0 10 2 0   bin kafka-configs sh --zookeeper  lt zk host gt  --alter --entity-type topics --entity-name test topic --add-config retention ms 86400000   Topic config alter operations have been deprecated for bin kafka-topics sh    WARNING  Altering topic configuration from this script has been deprecated and may be removed in future releases       Going forward  please use kafka-configs sh for this functionality

User · Answer

The correct config key is retention ms    bin kafka-topics sh --zookeeper zk prod yoursite com --alter --topic as-access --config retention ms 86400000 Updated config for topic  my-topic

User · Answer

I tested and used this command in kafka confluent V4 0 0 and  apache kafka V 1 0 0 and 1 0 1   opt kafka confluent-4 0 0 bin kafka-configs --zookeeper XX XX XX XX 2181 --entity-type topics --entity-name test --alter --add-config  retention ms 55000   test is the topic name   I think it works well in other versions too

[apache-kafka] changing kafka retention period during runtime

Examples related to apache-kafka

Examples related to retention