Extract data from log file in specified range of time

Question

I want to extract information from a log file using a shell script  bash  based on time range  A line in the log file looks like this   172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET   HTTP 1 1  200 123     Mozilla 5 0  compatible  Konqueror 2 2 2-2  Linux     i want to extract data specific intervals  For example I need to look only at the events which happened during the last X minutes or X days ago from the last recorded data  I m new in shell scripting but i have tried to use grep command

User · Answer

Use grep and regular expressions  for example if you want 4 minutes interval of logs   grep  31 Mar 2002 19 3 1-5   logfile   will return all logs lines between 19 31 and 19 35 on 31 Mar 2002  Supposing you need the last 5 days starting from today 27 Sep 2011 you may use the following   grep  2 3-7  Sep 2011  logfile

User · Answer

You can use sed for this  For example     sed -n   Feb 23 13 55   Feb 23 14 00 p   var log mail log Feb 23 13 55 01 messagerie postfix smtpd 20964   connect from localhost 127 0 0 1  Feb 23 13 55 01 messagerie postfix smtpd 20964   lost connection after CONNECT from localhost 127 0 0 1  Feb 23 13 55 01 messagerie postfix smtpd 20964   disconnect from localhost 127 0 0 1  Feb 23 13 55 01 messagerie pop3d  Connection  ip    ffff 127 0 0 1        How it works  The -n switch tells sed to not output each line of the file it reads  default behaviour    The last p after the regular expressions tells it to print lines that match the preceding expression   The expression   pattern1   pattern2   will print everything that is between first pattern and second pattern  In this case it will print every line it finds between the string Feb 23 13 55 and the string Feb 23 14 00   More info here

User · Answer

well  I have spent some time on your date format       however  finally i worked it out     let s take an example file  named logFile   i made it a bit short   say  you want to get last 5 mins  log in this file   172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 20 41  0200   GET      lines below are what you want  5 mins till the last record  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET    here is the solution     this variable you could customize  important is convert to seconds     e g 5days    5 24 3600   x    5 60      here we take 5 mins as example    this line get the timestamp in seconds of last line of your logfile last   tail -n1 logFile awk -F          gsub           2   sub          2    date   s -d     2     getline d  print d        this awk will give you lines you needs  awk -F       -v last  last -v x  x    gsub           2   sub          2    date   s -d     2     getline d  if  last-d lt  x print  0    logFile         output                       172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 27 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET  172 16 0 3 - -  31 Mar 2002 19 30 41  0200   GET   EDIT  you may notice that in the output the   and   are disappeared  If you do want them back  you can change the last awk line print  0 -  print  1      2      3

User · Answer

You can use this for getting current and log times      bin bash  log  log file name  while read line do   current hours  date   awk  BEGIN FS            print  4      current minutes  date   awk  BEGIN FS            print  5      current seconds  date   awk  BEGIN FS            print  6       log file hours  echo  line   awk  BEGIN FS              print   7      log file minutes  echo  line   awk  BEGIN FS              print   8      log file seconds  echo  line   awk  BEGIN FS              print   9        done  lt   log   And compare log file   and current   variables

User · Answer

I used this command to find last 5 minutes logs for particular event  DHCPACK   try below     grep  DHCPACK   var log messages   grep    date   h   d     date --date  5 min ago   H -  date   H

[bash] Extract data from log file in specified range of time

Examples related to bash