[java] How to configure log4j to only keep log files for the last seven days?

I have the following logging problem with several Java applications using log4j for logging:

I want log files to be rotated daily, like

log.2010-09-10
log.2010-09-09
log.2010-09-08
log.2010-09-07
log.2010-09-06
log.2010-09-05
log.2010-09-04

But for data security reasons we are not allowed to keep log files for longer than seven days at my company. So the generation of the next next log file log.2010-09-11 should trigger the deletion of log.2010-09-04. Is it possible to configure such a behaviour with log4j? If not, do you know another elegant solution for this kind of logging problem?

I create this Methode and call it by closing the application:

  public void deleteFiles(){

    File f = new File("log");
    File[] fileArray = f.listFiles();
    double timenow = System.currentTimeMillis();

    double olderTenDays = timenow - 864000000;// MS for ten days

    for (int i = 0; i < fileArray.length; i++) {

        if(fileArray[i].lastModified()< olderTenDays )
           fileArray[i].delete();
    }
 }

I assume you're using RollingFileAppender? In which case, it has a property called MaxBackupIndex which you can set to limit the number of files. For example:

log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=example.log
log4j.appender.R.MaxFileSize=100KB
log4j.appender.R.MaxBackupIndex=7
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n

There is another option DailyRollingFileAppender. but it lacks the auto delete (keep 7 days log) feature you looking for

sample

log4j.appender.DRF=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRF.File=example.log
log4j.appender.DRF.DatePattern='.'yyyy-MM-dd

I do come across something call org.apache.log4j.CompositeRollingAppender, which is combine both the features of the RollingFileAppender (maxSizeRollBackups, no. of backup file) and DailyRollingFileAppender (roll by day).

But have not tried that out, seems is not the standard 1.2 branch log4j feature.

The class DailyRollingFileAppender uses the DatePattern option to specify the rolling schedule. This pattern should follow the SimpleDateFormat conventions from Std. Ed. v1.4.2. So, we have to use E option (Day in week). For example:

<param name="DatePattern" value="'.'EEE"/>

See more about DailyRollingFileAppender class from log4j javadoc here. Unfortunately the Java 1.4.2 documentation is no longer online, but you can download a copy here.

According to the following post, you can't do this with log4j: Use MaxBackupIndex in DailyRollingFileAppender -log4j

As far as I know, this functionality was supposed to make it into log4j 2.0 but that effort got sidetracked. According to the logback website, logback is the intended successor to log4j so you might consider using that.

There's an API called SLF4J which provides a common API to logging. It will load up the actual logging implementation at runtime so depending on the configuration that you have provided, it might use java.util.log or log4j or logback or any other library capable of providing logging facilities. There'll be a bit of up-front work to go from using log4j directly to using SLF4J but they provide some tools to automate this process. Once you've converted your code to use SLF4J, switching logging backends should simply be a case of changing the config file.

My script based on @dogbane's answer

/etc/cron.daily/hbase

#!/bin/sh
find /var/log/hbase -type f -name "phoenix-hbase-server.log.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]" -exec bzip2 {} ";"
find /var/log/hbase -type f -regex ".*.out.[0-9][0-9]?" -exec bzip2 {} ";"
find /var/log/hbase -type f -mtime +7 -name "*.bz2" -exec rm -f {} ";"

/etc/cron.daily/tomcat

#!/bin/sh
find /opt/tomcat/log/ -type f -mtime +1 -name "*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].*log" -exec bzip2 {} ";"
find /opt/tomcat/log/ -type f -mtime +1 -name "*.[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9].txt" -exec bzip2 {} ";"
find /opt/tomcat/log/ -type f -mtime +7 -name "*.bz2" -exec rm -f {} ";"

because Tomcat rotate needs one day delay.

I had set:

log4j.appender.R=org.apache.log4j.DailyRollingFileAppender
log4j.appender.R.DatePattern='.'yyyy-MM-dd
# Archive log files (Keep one year of daily files)
log4j.appender.R.MaxBackupIndex=367

Like others before me, the DEBUG option showed me the error:

log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.DailyRollingFileAppender.

Here is an idea I have not tried yet, suppose I set the DatePattern such that the files overwrite each other after the required time period. To retain a year's worth I could try setting:

log4j.appender.R.DatePattern='.'MM-dd

Would it work or would it cause an error ? Like that it will take a year to find out, I could try:

log4j.appender.R.DatePattern='.'dd

but it will still take a month to find out.

If you are using Linux, you can configure a cron job using tmpwatch.

Most Linux systems have a tmpwatch cron job that cleans up the /tmp directory. You can add another that monitors your logging directory and deletes files over 7 days old.

If you are using a different system, there are probably equivalent utilities.

Use the setting log4j.appender.FILE.RollingPolicy.FileNamePattern, e.g. log4j.appender.FILE.RollingPolicy.FileNamePattern=F:/logs/filename.log.%d{dd}.gz for keeping logs one month before rolling over.

I didn't wait for one month to check but I tried with mm (i.e. minutes) and confirmed it overwrites, so I am assuming it will work for all patterns.

Inspite of starting a chrone job, for the task, we can use log4j2.properties file in config folder of logstash. Have a look at the link below, this will be helpful.

https://github.com/elastic/logstash/issues/7482

I came across this appender here that does what you want, it can be configured to keep a specific number of files that have been rolled over by date.

Download: http://www.simonsite.org.uk/download.htm

Example (groovy):

new TimeAndSizeRollingAppender(name: 'timeAndSizeRollingAppender',
   file: 'logs/app.log', datePattern: '.yyyy-MM-dd',
   maxRollFileCount: 7, compressionAlgorithm: 'GZ',
   compressionMinQueueSize: 2,
   layout: pattern(conversionPattern: "%d [%t] %-5p %c{2} %x - %m%n"))

There's also a DailyRollingFileAppender

Edit: after reading this worrying statement:

DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss. The log4j extras companion includes alternatives which should be considered for new deployments and which are discussed in the documentation for org.apache.log4j.rolling.RollingFileAppender.

from the above URL (which I never realized before), then the log4j-extras looks to be a better option.

log2j now has support to delete old logs.

Take a look at DefaultRolloverStrategy tag and at a snippet below.

It

• creates up to 10 archives on the same day,

• will parse the ${baseDir} directory that you define under the Properties tag at max depth of 2 with log filename matching "app-*.log.gz"

• delete logs older than 7 days but keep the most recent 5 logs if your most recent 5 logs are older than 7 days.

  <DefaultRolloverStrategy max="10">
    <Delete basePath="${baseDir}" maxDepth="2">
      <IfFileName glob="*/app-*.log.gz">
        <IfLastModified age="7d">
          <IfAny>
            <IfAccumulatedFileCount exceeds="5" />
          </IfAny>
        </IfLastModified>
      </IfFileName>
    </Delete>
  </DefaultRolloverStrategy>
  

A good debug option is if you set:

<Configuration status="trace">

and use testMode Option like this:

        <DefaultRolloverStrategy>
          <Delete basePath="${baseDir}" testMode="true">
            <IfFileName glob="*.log" />
            <IfLastModified age="7d" />
          </Delete>
        </DefaultRolloverStrategy>
        

You can see in console log what files would get deleted without deleting the files right away.