How to read a file in reverse order

Question

How to read a file in reverse order using python  I want to read a file from last line to first line

User · Answer

You can also use python module file read backwards   After installing it  via pip install file read backwards  v1 2 1   you can read the entire file backwards  line-wise  in a memory efficient manner via      usr bin env python2 7  from file read backwards import FileReadBackwards  with FileReadBackwards   path to file   encoding  utf-8   as frb      for l in frb           print l   It supports  utf-8   latin-1   and  ascii  encodings   Support is also available for python3  Further documentation can be found at http   file-read-backwards readthedocs io en latest readme html

User · Answer

def reverse lines filename       y open filename  readlines       return y   -1

User · Answer

If you are concerned about file size   memory usage  memory-mapping the file and scanning backwards for newlines is a solution   How to search for a string in text files

User · Answer

a simple function to create a second file reversed  linux only    import os def tac file1  file2        print os system  tac  s  gt   s     file1 file2      how to use  tac  ordered csv    reversed csv   f   open  reversed csv

User · Answer

import sys f   open sys argv 1     r   for line in f readlines     -1       print line

User · Answer

def previous line self  opened file           opened file seek 0  os SEEK END          position   opened file tell           buffer   bytearray           while position  gt   0              opened file seek position              position -  1             new byte   opened file read 1              if new byte    self NEW LINE                  parsed string   buffer decode                   yield parsed string                 buffer   bytearray               elif new byte    self EMPTY BYTE                  continue             else                  new byte array   bytearray new byte                  new byte array extend buffer                  buffer   new byte array         yield None   to use   opened file   open filepath   rb   iterator   self previous line opened file  line   next iterator   one step close opened file

User · Answer

Accepted answer won t work for cases with large files that won t fit in memory  which is not a rare case    As it was noted by others   srohde answer looks good  but it has next issues    openning file looks redundant  when we can pass file object  amp  leave it to user to decide in which encoding it should be read  even if we refactor to accept file object  it won t work for all encodings  we can choose file with utf-8 encoding and non-ascii contents like      pass buf size equal to 1 and will have  UnicodeDecodeError   utf8  codec can t decode byte 0xb9 in position 0  invalid start byte   of course text may be larger but buf size may be picked up so it ll lead to obfuscated error like above  we can t specify custom line separator  we can t choose to keep line separator    So considering all these concerns I ve written separate functions    one which works with byte streams  second one which works with text streams and delegates its underlying byte stream to the first one and decodes resulting lines    First of all let s define next utility functions   ceil division for making division with ceiling  in contrast with standard    division with floor  more info can be found in this thread   def ceil division left number  right number               Divides given numbers with ceiling              return - -left number    right number    split for splitting string by given separator from right end with ability to keep it   def split string  separator  keep separator               Splits given string by given separator              parts   string split separator      if keep separator           parts  last part   parts         parts    part   separator for part in parts          if last part              return parts    last part      return parts   read batch from end to read batch from the right end of binary stream  def read batch from end byte stream  size  end position               Reads batch from the end of given byte stream              if end position  gt  size          offset   end position - size     else          offset   0         size   end position     byte stream seek offset      return byte stream read size    After that we can define function for reading byte stream in reverse order like  import functools import itertools import os from operator import methodcaller  sub   def reverse binary stream byte stream  batch size None                            lines separator None                            keep lines separator True       if lines separator is None          lines separator    b  r   b  n   b  r n           lines splitter   methodcaller str splitlines   name                                          keep lines separator      else          lines splitter   functools partial split                                             separator lines separator                                             keep separator keep lines separator      stream size   byte stream seek 0  os SEEK END      if batch size is None          batch size   stream size or 1     batches count   ceil division stream size  batch size      remaining bytes indicator   itertools islice              itertools accumulate itertools chain  stream size                                                    itertools repeat batch size                                     sub               batches count      try          remaining bytes count   next remaining bytes indicator      except StopIteration          return      def read batch position           result   read batch from end byte stream                                       size batch size                                       end position position          while result startswith lines separator               try                  position   next remaining bytes indicator              except StopIteration                  break             result    read batch from end byte stream                                            size batch size                                            end position position                          result          return result      batch   read batch remaining bytes count      segment   lines   lines splitter batch      yield from reverse lines      for remaining bytes count in remaining bytes indicator          batch   read batch remaining bytes count          lines   lines splitter batch          if batch endswith lines separator               yield segment         else              lines -1     segment         segment   lines   lines         yield from reverse lines      yield segment   and finally a function for reversing text file can be defined like   import codecs   def reverse file file  batch size None                    lines separator None                   keep lines separator True       encoding   file encoding     if lines separator is not None          lines separator   lines separator encode encoding      yield from map functools partial codecs decode                                       encoding encoding                      reverse binary stream                             file buffer                             batch size batch size                             lines separator lines separator                             keep lines separator keep lines separator       Tests  Preparations  I ve generated 4 files using fsutil command    empty txt with no contents  size 0MB tiny txt with size of 1MB small txt with size of 10MB large txt with size of 50MB   also I ve refactored  srohde solution to work with file object instead of file path   Test script  from timeit import Timer  repeats count   7 number   1 create setup     from collections import deque n                   from   main   import reverse file  reverse readline n                   file   open         format srohde solution     with file  n                          deque reverse readline file  n                                                 buf size 8192                                  maxlen 0    azat ibrakov solution     with file  n                                deque reverse file file  n                                                   lines separator    n   n                                                   keep lines separator False  n                                                   batch size 8192   maxlen 0    print  reversing empty file by  srohde          min Timer srohde solution                  create setup  empty txt    repeat repeats count  number    print  reversing empty file by  Azat Ibrakov          min Timer azat ibrakov solution                  create setup  empty txt    repeat repeats count  number    print  reversing tiny file  1MB  by  srohde          min Timer srohde solution                  create setup  tiny txt    repeat repeats count  number    print  reversing tiny file  1MB  by  Azat Ibrakov          min Timer azat ibrakov solution                  create setup  tiny txt    repeat repeats count  number    print  reversing small file  10MB  by  srohde          min Timer srohde solution                  create setup  small txt    repeat repeats count  number    print  reversing small file  10MB  by  Azat Ibrakov          min Timer azat ibrakov solution                  create setup  small txt    repeat repeats count  number    print  reversing large file  50MB  by  srohde          min Timer srohde solution                  create setup  large txt    repeat repeats count  number    print  reversing large file  50MB  by  Azat Ibrakov          min Timer azat ibrakov solution                  create setup  large txt    repeat repeats count  number      Note  I ve used collections deque class to exhaust generator   Outputs  For PyPy 3 5 on Windows 10   reversing empty file by  srohde  8 31e-05 reversing empty file by  Azat Ibrakov  0 00016090000000000028 reversing tiny file  1MB  by  srohde  0 160081 reversing tiny file  1MB  by  Azat Ibrakov  0 09594989999999998 reversing small file  10MB  by  srohde  8 8891863 reversing small file  10MB  by  Azat Ibrakov  5 323388100000001 reversing large file  50MB  by  srohde  186 5338368 reversing large file  50MB  by  Azat Ibrakov  99 07450229999998   For CPython 3 5 on Windows 10   reversing empty file by  srohde  3 600000000000001e-05 reversing empty file by  Azat Ibrakov  4 519999999999958e-05 reversing tiny file  1MB  by  srohde  0 01965560000000001 reversing tiny file  1MB  by  Azat Ibrakov  0 019207699999999994 reversing small file  10MB  by  srohde  3 1341862999999996 reversing small file  10MB  by  Azat Ibrakov  3 0872588000000007 reversing large file  50MB  by  srohde  82 01206720000002 reversing large file  50MB  by  Azat Ibrakov  82 16775059999998   So as we can see it performs like original solution  but is more general and free of its disadvantages listed above     Advertisement  I ve added this to 0 3 0 version of lz package  requires Python 3 5   that have many well-tested functional iterating utilities   Can be used like   import io  from lz iterating import reverse       with open  path to file   as file       for line in reverse file  batch size io DEFAULT BUFFER SIZE            print line    It supports all standard encodings  maybe except utf-7 since it is hard for me to define a strategy for generating strings encodable with it

User · Answer

for line in reversed open  filename   readlines         print line rstrip     And in Python 3   for line in reversed list open  filename          print line rstrip

User · Answer

I had to do this some time ago and used the below code  It pipes to the shell  I am afraid i do not have the complete script anymore  If you are on a unixish operating system  you can use  tac   however on e g  Mac OSX tac command does not work  use tail -r  The below code snippet tests for which platform you re on  and adjusts the command accordingly    We need a command to reverse the line order of the file  On Linux this   is  tac   on OSX it is  tail -r     tac  is not supported on osx   tail -r  is not supported on linux   if sys platform     darwin       command      tail -r  elif sys platform     linux2       command      tac  else      raise EnvironmentError  Platform  s not supported    sys platform

User · Answer

import re  def filerev somefile  buffer 0x20000     somefile seek 0  os SEEK END    size   somefile tell     lines          rem   size   buffer   pos   max 0   size    buffer - 1    buffer    while pos  gt   0      somefile seek pos  os SEEK SET      data   somefile read rem   buffer    lines 0      rem   0     lines   re findall     n   n    data      ix   len lines  - 2     while ix  gt  0        yield lines ix        ix -  1     pos -  buffer   else      yield lines 0   with open sys argv 1    r   as f    for line in filerev f       sys stdout write line

User · Answer

you would need to first open your file in read format  save it to a variable  then open the second file in write format where you would write or append the variable using a the    -1  slice  completely reversing the file  You can also use readlines   to make it into a list of lines  which you can manipulate  def copy and reverse filename  newfile       with open filename  as file          text   file read       with open newfile   w   as file2          file2 write text   -1

User · Answer

How about something like this   import os   def readlines reverse filename       with open filename  as qfile          qfile seek 0  os SEEK END          position   qfile tell           line              while position  gt   0              qfile seek position              next char   qfile read 1              if next char      n                   yield line   -1                  line                  else                  line    next char             position -  1         yield line   -1    if   name         main         for qline in readlines reverse raw input             print qline   Since the file is read character by character in reverse order  it will work even on very large files  as long as individual lines fit into memory

User · Answer

for line in reversed open  file   readlines         print line rstrip     If you are on linux  you can use tac command     tac file   2 recipes you can find in ActiveState here and here

User · Answer

Read the file line by line and then add it on a list in reverse order   Here is an example of code    reverse      with open  file txt    r   as file      for line in file          line   line strip            reverse 0 0    line

User · Answer

Always use with when working with files as it handles everything for you   with open  filename    r   as f      for line in reversed f readlines             print line   Or in Python 3   with open  filename    r   as f      for line in reversed list f readlines              print line

User · Answer

I don t think this has been mentioned before  but using deque from collections and reverse works for me  from collections import deque  fs   open  quot test txt quot   quot rU quot   fr   deque fs  fr reverse      reverse in-place  returns None  for li in fr     print li  fs close

User · Answer

Most of the answers need to read the whole file before doing anything  This sample reads increasingly large samples from the end   I only saw Murat Y  kselen s answer while writing this answer  It s nearly the same  which I suppose is a good thing  The sample below also deals with  r and increases its buffersize at each step  I also have some unit tests to back this code up   def readlines reversed f           Iterate over the lines in a file in reverse  The file must be     open in  rb  mode  Yields the lines unencoded  as bytes   including the     newline character  Produces the same result as readlines  but reversed      If this is used to reverse the line in a file twice  the result is     exactly the same              head   b       f seek 0  2      t   f tell       buffersize  maxbuffersize   64  4096     while True          if t  lt   0              break           Read next block         buffersize   min buffersize   2  maxbuffersize          tprev   t         t   max 0  t - buffersize          f seek t          lines   f read tprev - t  splitlines True            Align to line breaks         if not lines -1  endswith  b  n   b  r                 lines -1     head    current tail is previous head         elif head    b  n  and lines -1  endswith b  r                lines -1     head    Keep  r n together         elif head              lines append head          head   lines pop 0     can be   n   ok            Iterate over current block in reverse         for line in reversed lines               yield line     if head          yield head

User · Answer

Thanks for the answer  srohde  It has a small bug checking for newline character with  is  operator  and I could not comment on the answer with 1 reputation  Also I d like to manage file open outside because that enables me to embed my ramblings for luigi tasks   What I needed to change has the form   with open filename  as fp      for line in fp           print line     contains new line         print   gt    lt   format line    I d love to change to   with open filename  as fp      for line in reversed fp iter fp  4            print line     contains new line         print   gt    lt   format line    Here is a modified answer that wants a file handle and keeps newlines   def reversed fp iter fp  buf size 8192          a generator that returns the lines of a file in reverse order     ref  https   stackoverflow com a 23646049 8776239             segment   None    holds possible incomplete segment at the beginning of the buffer     offset   0     fp seek 0  os SEEK END      file size   remaining size   fp tell       while remaining size  gt  0          offset   min file size  offset   buf size          fp seek file size - offset          buffer   fp read min remaining size  buf size           remaining size -  buf size         lines   buffer splitlines True            the first line of the buffer is probably not a complete line so           we ll save it and append it to the last line of the next buffer           we read         if segment is not None                if the previous chunk starts right from the beginning of line               do not concat the segment to the last line of new chunk               instead  yield the segment first             if buffer -1       n                    print  buffer ends with newline                  yield segment             else                  lines -1     segment                  print  enlarged last line to  gt    lt   len     format lines -1   len lines           segment   lines 0          for index in range len lines  - 1  0  -1               if len lines index                    yield lines index        Don t yield None if the file was empty     if segment is not None          yield segment

User · Answer

A correct  efficient answer written as a generator   import os  def reverse readline filename  buf size 8192          A generator that returns the lines of a file in reverse order        with open filename  as fh          segment   None         offset   0         fh seek 0  os SEEK END          file size   remaining size   fh tell           while remaining size  gt  0              offset   min file size  offset   buf size              fh seek file size - offset              buffer   fh read min remaining size  buf size               remaining size -  buf size             lines   buffer split   n                 The first line of the buffer is probably not a complete line so               we ll save it and append it to the last line of the next buffer               we read             if segment is not None                    If the previous chunk starts right from the beginning of line                   do not concat the segment to the last line of new chunk                    Instead  yield the segment first                  if buffer -1       n                       lines -1     segment                 else                      yield segment             segment   lines 0              for index in range len lines  - 1  0  -1                   if lines index                       yield lines index            Don t yield None if the file was empty         if segment is not None              yield segment

User · Answer

with open  filename   as f       print f read     -1

User · Answer

Here you can find my my implementation  you can limit the ram usage by changing the  buffer  variable  there is a bug that the program prints an empty line in the beginning   And also ram usage may be increase if there is no new lines for more than buffer bytes   leak  variable will increase until seeing a new line    n     This is also working for 16 GB files which is bigger then my total memory   import os sys buffer   1024 1024   1MB f   open sys argv 1   f seek 0  os SEEK END  filesize   f tell    division  remainder   divmod filesize  buffer  line leak     for chunk counter in range 1 division   2       if division - chunk counter  lt  0          f seek 0  os SEEK SET          chunk   f read remainder      elif division - chunk counter  gt   0          f seek - buffer chunk counter   os SEEK END          chunk   f read buffer       chunk lines reversed   list reversed chunk split   n         if line leak    add line leak from previous chunk to beginning         chunk lines reversed 0     line leak        after reversed  save the leakedline for next chunk iteration     line leak   chunk lines reversed pop        if chunk lines reversed          print   n  join chunk lines reversed        print the last leaked line     if division - chunk counter  lt  0          print line leak

[python] How to read a file in reverse order?

Examples related to python

Examples related to file

Examples related to reverse