Get last n lines of a file similar to tail

Question

I m writing a log file viewer for a web application and for that I want to paginate through the lines of the log file   The items in the file are line based with the newest item at the bottom  So I need a tail   method that can read n lines from the bottom and support an offset   This is hat I came up with  def tail f  n  offset 0        quot  quot  quot Reads a n lines from f with an offset of offset lines  quot  quot  quot      avg line length   74     to read   n   offset     while 1          try              f seek - avg line length   to read   2          except IOError                woops   apparently file is smaller than what we want               to step back  go to the beginning instead             f seek 0          pos   f tell           lines   f read   splitlines           if len lines   gt   to read or pos    0              return lines -to read offset and -offset or None          avg line length    1 3  Is this a reasonable approach   What is the recommended way to tail log files with offsets

User · Answer

I had to read a specific value from the last line of a file, and stumbled upon this thread. Rather than reinventing the wheel in Python, I ended up with a tiny shell script, saved as /usr/local/bin/get_last_netp:

#! /bin/bash
tail -n1 /home/leif/projects/transfer/export.log | awk {'print $14'}

And in the Python program:

from subprocess import check_output

last_netp = int(check_output("/usr/local/bin/get_last_netp"))

User · Answer

Update for answer given by A Coady  Works with python 3   This uses Exponential Search and will buffer only N lines from back and is very efficient   import time import os import sys  def tail f  n       assert n  gt   0     pos  lines   n 1            set file pointer to end      f seek 0  os SEEK END       isFileSmall   False      while len lines   lt   n          try              f seek f tell   - pos  os SEEK SET          except ValueError as e                lines greater than file seeking size               seek to start             f seek 0 os SEEK SET              isFileSmall   True         except IOError              print  Some problem reading seeking the file               sys exit -1          finally              lines   f readlines               if isFileSmall                  break          pos    2      print lines       return lines -n       with open  stream logs txt   as f      while True           time sleep 0 5          print tail f 2

User · Answer

The simplest way is to use deque   from collections import deque  def tail filename  n 10       with open filename  as f          return deque f  n

User · Answer

For efficiency with very large files  common in logfile situations where you may want to use tail   you generally want to avoid reading the whole file  even if you do do it without reading the whole file into memory at once   However  you do need to somehow work out the offset in lines rather than characters   One possibility is reading backwards with seek   char by char  but this is very slow   Instead  its better to process in larger blocks   I ve a utility function I wrote a while ago to read files backwards that can be used here   import os  itertools  def rblocks f  blocksize 4096          Read file as series of blocks from end of file to start       The data itself is in normal order  only the order of the blocks is reversed      ie   hello world  - gt    ld   wor    lo     hel       Note that the file must be opened in binary mode              if  b  not in f mode lower            raise Exception  File must be opened using binary mode        size   os stat f name  st size     fullblocks  lastblock   divmod size  blocksize         The first end of file  block will be short  since this leaves        the rest aligned on a blocksize boundary   This may be more        efficient than having the last  first in file  block be short     f seek -lastblock 2      yield f read lastblock       for i in range fullblocks-1 -1  -1           f seek i   blocksize          yield f read blocksize   def tail f  nlines       buf          result          for block in rblocks f           buf   block   buf         lines   buf splitlines              Return all lines except the first  since may be partial          if lines              result extend lines 1      First line may not be complete             if len result   gt   nlines                   return result -nlines                buf   lines 0       return   buf  result  -nlines     f open  file to tail txt   rb   for line in tail f  20       print line    Edit  Added more specific version  avoids need to reverse twice

User · Answer

Although this isn t really on the efficient side with big files  this code is pretty straight-forward    It reads the file object  f  It splits the string returned using newlines   n  It gets the array lists last indexes  using the negative sign to stand for the last indexes  and the   to get a subarray   def tail f n       return   n  join f read   split   n   -n

User · Answer

There are some existing implementations of tail on pypi which you can install using pip     mtFileUtil multitail log4tailer       Depending on your situation  there may be advantages to using one of these existing tools

User · Answer

Simple        with open  test txt   as f  data   f readlines   tail   data -2   print    join tail

User · Answer

Several of these solutions have issues if the file doesn t end in  n or in ensuring the complete first line is read   def tail file  n 1  bs 1024       f   open file      f seek -1 2      l   1-f read 1  count   n     If file doesn t end in  n  count it anyway      B   f tell       while n  gt   l and B  gt  0              block   min bs  B              B -  block             f seek B  0              l    f read block  count   n       f seek B  0      l   min l n    discard first  incomplete  line if l  gt  n     lines   f readlines   -l       f close       return lines

User · Answer

If reading the whole file is acceptable then use a deque   from collections import deque deque f  maxlen n    Prior to 2 6  deques didn t have a maxlen option  but it s easy enough to implement   import itertools def maxque items  size       items   iter items      q   deque itertools islice items  size       for item in items          del q 0          q append item      return q   If it s a requirement to read the file from the end  then use a gallop  a k a exponential  search   def tail f  n       assert n  gt   0     pos  lines   n 1         while len lines   lt   n          try              f seek -pos  2          except IOError              f seek 0              break         finally              lines   list f          pos    2     return lines -n

User · Answer

S Lott s answer above almost works for me but ends up giving me partial lines  It turns out that it corrupts data on block boundaries because data holds the read blocks in reversed order  When    join data  is called  the blocks are in the wrong order  This fixes that   def tail f  window 20               Returns the last  window  lines of file  f  as a list      f - a byte file-like object             if window    0          return        BUFSIZ   1024     f seek 0  2      bytes   f tell       size   window   1     block   -1     data          while size  gt  0 and bytes  gt  0          if bytes - BUFSIZ  gt  0                Seek back one whole BUFSIZ             f seek block   BUFSIZ  2                read BUFFER             data insert 0  f read BUFSIZ           else                file too small  start from begining             f seek 0 0                only read what was not read             data insert 0  f read bytes           linesFound   data 0  count   n           size -  linesFound         bytes -  BUFSIZ         block -  1     return    join data  splitlines   -window

User · Answer

Here is my answer  Pure python  Using timeit it seems pretty fast  Tailing 100 lines of a log file that has 100 000 lines    gt  gt  gt  timeit timeit  tail tail f  100  4098     import tail  f   open  log txt    r      number 10  0 0014600753784179688  gt  gt  gt  timeit timeit  tail tail f  100  4098     import tail  f   open  log txt    r      number 100  0 00899195671081543  gt  gt  gt  timeit timeit  tail tail f  100  4098     import tail  f   open  log txt    r      number 1000  0 05842900276184082  gt  gt  gt  timeit timeit  tail tail f  100  4098     import tail  f   open  log txt    r      number 10000  0 5394978523254395  gt  gt  gt  timeit timeit  tail tail f  100  4098     import tail  f   open  log txt    r      number 100000  5 377126932144165   Here is the code   import os   def tail f  lines 1   buffer 4098          Tail a file and get X lines from the end          place holder for the lines found     lines found             block counter will be multiplied by buffer       to get the block size from the end     block counter   -1        loop until we find X lines     while len lines found   lt  lines          try              f seek block counter    buffer  os SEEK END          except IOError     either file is too small  or too many lines requested             f seek 0              lines found   f readlines               break          lines found   f readlines              we found enough lines  get out           Removed this line because it was redundant the while will catch           it  I left it for history           if len lines found   gt  lines               break            decrement the block counter to get the           next X bytes         block counter -  1      return lines found -lines

User · Answer

import itertools fname    log txt  offset   5 n   10 with open fname  as f      n last lines   list reversed  x for x in itertools islice f  None   - offset 1  - offset n 1  -1

User · Answer

This may be quicker than yours   Makes no assumptions about line length   Backs through the file one block at a time till it s found the right number of   n  characters   def tail  f  lines 20        total lines wanted   lines      BLOCK SIZE   1024     f seek 0  2      block end byte   f tell       lines to go   total lines wanted     block number   -1     blocks        blocks of size BLOCK SIZE  in reverse order starting                   from the end of the file     while lines to go  gt  0 and block end byte  gt  0          if  block end byte - BLOCK SIZE  gt  0                 read the last block we haven t yet read             f seek block number BLOCK SIZE  2              blocks append f read BLOCK SIZE           else                file too small  start from begining             f seek 0 0                only read what was not read             blocks append f read block end byte           lines found   blocks -1  count   n           lines to go -  lines found         block end byte -  BLOCK SIZE         block number -  1     all read text      join reversed blocks       return   n  join all read text splitlines   -total lines wanted      I don t like tricky assumptions about line length when -- as a practical matter -- you can never know things like that   Generally  this will locate the last 20 lines on the first or second pass through the loop   If your 74 character thing is actually accurate  you make the block size 2048 and you ll tail 20 lines almost immediately   Also  I don t burn a lot of brain calories trying to finesse alignment with physical OS blocks   Using these high-level I O packages  I doubt you ll see any performance consequence of trying to align on OS block boundaries   If you use lower-level I O  then you might see a speedup     UPDATE  for Python 3 2 and up  follow the process on bytes as In text files  those opened without a  b  in the mode string   only seeks relative to the beginning of the file are allowed  the exception being seeking to the very file end with seek 0  2      eg  f    open  C         apache logs txt    rb     def tail f  lines 20       total lines wanted   lines      BLOCK SIZE   1024     f seek 0  2      block end byte   f tell       lines to go   total lines wanted     block number   -1     blocks          while lines to go  gt  0 and block end byte  gt  0          if  block end byte - BLOCK SIZE  gt  0               f seek block number BLOCK SIZE  2              blocks append f read BLOCK SIZE           else              f seek 0 0              blocks append f read block end byte           lines found   blocks -1  count b  n           lines to go -  lines found         block end byte -  BLOCK SIZE         block number -  1     all read text   b   join reversed blocks       return b  n  join all read text splitlines   -total lines wanted

User · Answer

An even cleaner python3 compatible version that doesn t insert but appends  amp  reverses   def tail f  window 1               Returns the last  window  lines of file  f  as a list of bytes              if window    0          return b       BUFSIZE   1024     f seek 0  2      end   f tell       nlines   window   1     data          while nlines  gt  0 and end  gt  0          i   max 0  end - BUFSIZE          nread   min end  BUFSIZE           f seek i          chunk   f read nread          data append chunk          nlines -  chunk count b  n           end -  nread     return b  n  join b   join reversed data   splitlines   -window      use it like this   with open path   rb   as f      last lines   tail f  3  decode  utf-8

User · Answer

The code I ended up using   I think this is the best so far   def tail f  n  offset None          Reads a n lines from f with an offset of offset lines   The return     value is a tuple in the form    lines  has more    where  has more  is     an indicator that is  True  if there are more lines in the file              avg line length   74     to read   n    offset or 0       while 1          try              f seek - avg line length   to read   2          except IOError                woops   apparently file is smaller than what we want               to step back  go to the beginning instead             f seek 0          pos   f tell           lines   f read   splitlines           if len lines   gt   to read or pos    0              return lines -to read offset and -offset or None                        len lines   gt  to read or pos  gt  0         avg line length    1 3

User · Answer

Assumes a unix-like system on Python 2 you can do   import os def tail f  n  offset 0     stdin stdout   os popen2  tail -n   n offset     f    stdin close     lines   stdout readlines    stdout close     return lines   -offset    For python 3 you may do   import subprocess def tail f  n  offset 0       proc   subprocess Popen   tail    -n   n   offset  f   stdout subprocess PIPE      lines   proc stdout readlines       return lines    -offset

User · Answer

I found a probably the easiest way to find the first or last N lines of a file  Last N lines of a file For Ex N 10   file open  xyz txt   r   liner file readlines   for ran in range  len liner -N  len liner        print liner ran    First N lines of a file For Ex N 10   file open  xyz txt   r   liner file readlines   for ran in range 0 N 1       print liner ran

User · Answer

I found the Popen above to be the best solution  It s quick and dirty and it works For python 2 6 on Unix machine i used the following  def GetLastNLines self  n  fileName               Name            Get LastNLines     Description         Gets last n lines using Unix tail     Output          returns last n lines of a file     Keyword argument      n -- number of last lines to return     filename -- Name of the file you need to tail into             p   subprocess Popen   tail   -n  str n  self   fileName   stdout subprocess PIPE      soutput  sinput   p communicate       return soutput   soutput will have will contain last n lines of the code  to iterate through soutput line by line do   for line in GetLastNLines 50  myfile log   split   n        print line

User · Answer

you can go to the end of your file with f seek 0  2  and then read off lines one by one with the following replacement for readline     def readline backwards self  f       backline          last          while not last      n           backline   last   backline         if f tell    lt   0              return backline         f seek -1  1          last   f read 1          f seek -1  1      backline   last     last          while not last      n           backline   last   backline         if f tell    lt   0              return backline         f seek -1  1          last   f read 1          f seek -1  1      f seek 1  1      return backline

User · Answer

import time  attemps   600 wait sec   5 fname    YOUR PATH   with open fname   r   as f      where   f tell       for i in range attemps           line   f readline           if not line              time sleep wait sec              f seek where          else              print line    already has newline

User · Answer

abc    2018-06-16 04 45 18 68  filename    abc txt  with open filename  as myFile      for num  line in enumerate myFile  1           if abc in line              lastline   num print  last occurance of work at file is in   str lastline

User · Answer

Based on Eyecue answer  Jun 10  10 at 21 28   this class add head   and tail   method to file object   class File file       def head self  lines 2find 1           self seek 0                              Rewind file         return  self next   for x in xrange lines 2find        def tail self  lines 2find 1             self seek 0  2                           go to end of file         bytes in file   self tell                        lines found  total bytes scanned   0  0         while  lines 2find 1  gt  lines found and                bytes in file  gt  total bytes scanned                byte block   min 1024  bytes in file-total bytes scanned              self seek - byte block total bytes scanned   2              total bytes scanned    byte block             lines found    self read 1024  count   n           self seek -total bytes scanned  2          line list   list self readlines            return line list -lines 2find     Usage   f   File  path to file    r   f head 3  f tail 3

User · Answer

it s so simple   def tail fname nl   with open fname  as f      data f readlines    readlines return a list     print    join data -nl

User · Answer

If reading the whole file is acceptable then use a deque   from collections import deque deque f  maxlen n    Prior to 2 6  deques didn t have a maxlen option  but it s easy enough to implement   import itertools def maxque items  size       items   iter items      q   deque itertools islice items  size       for item in items          del q 0          q append item      return q   If it s a requirement to read the file from the end  then use a gallop  a k a exponential  search   def tail f  n       assert n  gt   0     pos  lines   n 1         while len lines   lt   n          try              f seek -pos  2          except IOError              f seek 0              break         finally              lines   list f          pos    2     return lines -n

User · Answer

Not the first example using a deque  but a simpler one   This one is general   it works on any iterable object  not just a file      usr bin env python import sys import collections def tail iterable  N       deq   collections deque       for thing in iterable          if len deq   gt   N              deq popleft           deq append thing      for thing in deq          yield thing if   name         main         for line in tail sys stdin 10           sys stdout write line

User · Answer

Update  papercrane solution to python3  Open the file with open filename   rb   and   def tail f  window 20          Returns the last  window  lines of file  f  as a list              if window    0          return         BUFSIZ   1024     f seek 0  2      remaining bytes   f tell       size   window   1     block   -1     data           while size  gt  0 and remaining bytes  gt  0          if remaining bytes - BUFSIZ  gt  0                Seek back one whole BUFSIZ             f seek block   BUFSIZ  2                read BUFFER             bunch   f read BUFSIZ          else                file too small  start from beginning             f seek 0  0                only read what was not read             bunch   f read remaining bytes           bunch   bunch decode  utf-8           data insert 0  bunch          size -  bunch count   n           remaining bytes -  BUFSIZ         block -  1      return    join data  splitlines   -window

User · Answer

This is my version of tailf  import sys  time  os  filename    path to file   try      with open filename  as f          size   os path getsize filename          if size  lt  1024              s   size         else              s   999         f seek -s  2          l   f read           print l         while True              line   f readline               if not line                  time sleep 1                  continue             print line except IOError      pass

User · Answer

Another Solution  if your txt file looks like this  mouse snake cat lizard wolf dog  you could reverse this file by simply using array indexing in python      contents    def tail contents n       with open  file txt   as file          for i in file readlines                contents append i       for i in contents  n -1           print i   tail contents -5    result   dog  wolf lizard cat

User · Answer

Simple and fast solution with mmap   import mmap import os  def tail filename  n          Returns last n lines from the filename  No exception handling        size   os path getsize filename      with open filename   rb   as f            for Windows the mmap parameters are different         fm   mmap mmap f fileno    0  mmap MAP SHARED  mmap PROT READ          try              for i in xrange size - 1  -1  -1                   if fm i       n                       n -  1                     if n    -1                          break             return fm i   1 if i else 0   splitlines           finally              fm close

User · Answer

On second thought  this is probably just as fast as anything here   def tail  f  window 20        lines       window     count  0     for l in f          lines count window   l         count    1     print lines count window    lines  count window    It s a lot simpler   And it does seem to rip along at a good pace

User · Answer

Here is a pretty simple implementation   with open   etc passwd    r   as f    try      f seek 0 2      s          while s count   n    lt  11        cur   f tell         f seek  cur - 10         s   f read 10    s       f seek  cur - 10       print s   except Exception as e      f readlines

User · Answer

based on S Lott s top voted answer  Sep 25  08 at 21 43   but fixed for small files   def tail the file  lines 2find 20         the file seek 0  2                           go to end of file     bytes in file   the file tell                    lines found  total bytes scanned   0  0     while lines 2find 1  gt  lines found and bytes in file  gt  total bytes scanned           byte block   min 1024  bytes in file-total bytes scanned          the file seek - byte block total bytes scanned   2          total bytes scanned    byte block         lines found    the file read 1024  count   n       the file seek -total bytes scanned  2      line list   list the file readlines        return line list -lines 2find         we read at least 21 line breaks from the bottom  block by block for speed      21 to ensure we don t get a half line   Hope this is useful

User · Answer

Posting an answer at the behest of commenters on my answer to a similar question where the same technique was used to mutate the last line of a file  not just get it   For a file of significant size  mmap is the best way to do this  To improve on the existing mmap answer  this version is portable between Windows and Linux  and should run faster  though it won t work without some modifications on 32 bit Python with files in the GB range  see the other answer for hints on handling this  and for modifying to work on Python 2    import io    Gets consistent version of open for both Py2 7 and Py3 x import itertools import mmap  def skip back lines mm  numlines  startidx          Factored out to simplify handling of n and offset        for   in itertools repeat None  numlines           startidx   mm rfind b  n   0  startidx          if startidx  lt  0              break     return startidx  def tail f  n  offset 0         Reopen file in binary mode     with io open f name   rb   as binf  mmap mmap binf fileno    0  access mmap ACCESS READ  as mm            len mm  - 1 handles files ending w newline by getting the prior line         startofline   skip back lines mm  offset  len mm  - 1          if startofline  lt  0              return       Offset lines consumed whole file  nothing to return               If using a generator function  yield-ing  see below                 this should be a plain return  no empty list          endoflines   startofline   1    Slice end to omit offset lines            Find start of lines to capture  add 1 to move from newline to beginning of following line          startofline   skip back lines mm  n  startofline    1            Passing True to splitlines makes it return the list of lines without           removing the trailing newline  if any   so list mimics f readlines           return mm startofline endoflines  splitlines True            If Windows style  r n newlines need to be normalized to  n  and input           is ASCII compatible  can normalize newlines with            return mm startofline endoflines  replace os linesep encode  ascii    b  n   splitlines True    This assumes the number of lines tailed is small enough you can safely read them all into memory at once  you could also make this a generator function and manually read a line at a time by replacing the final line with           mm seek startofline            Call mm readline n times  or until EOF  whichever comes first           Python 3 2 and earlier          for line in itertools islice iter mm readline  b     n               yield line            3 3           yield from itertools islice iter mm readline  b     n    Lastly  this read in binary mode  necessary to use mmap  so it gives str lines  Py2  and bytes lines  Py3   if you want unicode  Py2  or str  Py3   the iterative approach could be tweaked to decode for you and or fix newlines           lines   itertools islice iter mm readline  b     n          if f encoding     Decode if the passed file was opened with a specific encoding             lines    line decode f encoding  for line in lines          if  b  not in f mode     Fix line breaks if passed file opened in text mode             lines    line replace os linesep    n   for line in lines            Python 3 2 and earlier          for line in lines              yield line           3 3           yield from lines   Note  I typed this all up on a machine where I lack access to Python to test  Please let me know if I typoed anything  this was similar enough to my other answer that I think it should work  but the tweaks  e g  handling an offset  could lead to subtle errors  Please let me know in the comments if there are any mistakes

User · Answer

There is very useful module that can do this   from file read backwards import FileReadBackwards  with FileReadBackwards   tmp file   encoding  utf-8   as frb     getting lines by lines starting from the last line up for l in frb      print l

[python] Get last n lines of a file, similar to tail

Examples related to python

Examples related to file

Examples related to file-io

Examples related to tail

Examples related to logfile