Read large files in Java

Question

I need the advice from someone who knows Java very well and the memory issues  I have a large file  something like 1 5GB  and I need to cut this file in many  100 small files for example  smaller files   I know generally how to do it  using a BufferedReader   but I would like to know if you have any advice regarding the memory  or tips how to do it faster   My file contains text  it is not binary and I have about 20 character per line

User · Answer

Does it have to be done in Java  I e  does it need to be platform independent  If not  I d suggest using the  split  command in  nix  If you really wanted  you could execute this command via your java program  While I haven t tested  I imagine it perform faster than whatever Java IO implementation you could come up with

User · Answer

Yes   I also think that using read   with arguments like read Char    int init  int end  is a better way to read a such a large file  Eg   read buffer 0 buffer length    And I also experienced the problem of missing values of using the BufferedReader instead of BufferedInputStreamReader for a binary data input stream  So  using the BufferedInputStreamReader is a much better in this like case

User · Answer

Don t use read without arguments  It s very slow  Better read it to buffer and move it to file quickly   Use bufferedInputStream because it supports binary reading   And it s all

User · Answer

First  if your file contains binary data  then using BufferedReader would be a big mistake  because you would be converting the data to String  which is unnecessary and could easily corrupt the data   you should use a BufferedInputStream instead  If it s text data and you need to split it along linebreaks  then using BufferedReader is OK  assuming the file contains lines of a sensible length    Regarding memory  there shouldn t be any problem if you use a decently sized buffer  I d use at least 1MB to make sure the HD is doing mostly sequential reading and writing    If speed turns out to be a problem  you could have a look at the java nio packages - those are supposedly faster than java io

User · Answer

To save memory  do not unnecessarily store duplicate the data in memory  i e  do not assign them to variables outside the loop   Just process the output immediately as soon as the input comes in   It really doesn t matter whether you re using BufferedReader or not  It will not cost significantly much more memory as some implicitly seem to suggest  It will at highest only hit a few   from performance  The same applies on using NIO  It will only improve scalability  not memory use  It will only become interesting when you ve hundreds of threads running on the same file   Just loop through the file  write every line immediately to other file as you read in  count the lines and if it reaches 100  then switch to next file  etcetera   Kickoff example   String encoding    UTF-8   int maxlines   100  BufferedReader reader   null  BufferedWriter writer   null   try       reader   new BufferedReader new InputStreamReader new FileInputStream   bigfile txt    encoding        int count   0      for  String line   line   reader readLine       null             if  count     maxlines    0                close writer               writer   new BufferedWriter new OutputStreamWriter new FileOutputStream   smallfile     count   maxlines      txt    encoding                      writer write line           writer newLine            finally       close writer       close reader

User · Answer

You can use java nio which is faster than classical Input Output stream   http   java sun com javase 6 docs technotes guides io index html

User · Answer

This is a very good article  http   java sun com developer technicalArticles Programming PerfTuning   In summary  for great performance  you should    Avoid accessing the disk  Avoid accessing the underlying operating system  Avoid method calls  Avoid processing bytes and characters individually    For example  to reduce the access to disk  you can use a large buffer  The article describes various approaches

User · Answer

You can consider using memory-mapped files  via FileChannels    Generally a lot faster for large files   There are performance trade-offs that could make it slower  so YMMV   Related answer  Java NIO FileChannel versus FileOutputstream performance   usefulness

User · Answer

Unless you accidentally read in the whole input file instead of reading it line by line  then your primary limitation will be disk speed    You may want to try starting with a file containing 100 lines and write it to 100 different files one line in each and make the triggering mechanism work on the number of lines written to the current file   That program will be easily scalable to your situation

User · Answer

x000D   x000D  package all is well  x000D  import java io IOException  x000D  import java io RandomAccessFile  x000D  import java util concurrent ExecutorService  x000D  import java util concurrent Executors  x000D  import junit framework TestCase  x000D   x000D      x000D      author Naresh Bhabat x000D      x000D  Following  implementation helps to deal with extra large files in java  x000D  This program is tested for dealing with 2GB input file  x000D  There are some points where extra logic can be added in future  x000D   x000D   x000D  Pleasenote  if we want to deal with binary input file  then instead of reading line we need to read bytes from read file object  x000D   x000D   x000D   x000D  It uses random access file which is almost like streaming API  x000D   x000D   x000D                                              x000D  Notes regarding executor framework and its readings  x000D  Please note  ExecutorService executor   Executors newFixedThreadPool 10   x000D   x000D          for 10 threads Total time required for reading and writing the text in x000D              seconds 349 317 x000D      x000D             For 100 Total time required for reading the text and writing     seconds 464 042 x000D      x000D             For 1000   Total time required for reading and writing text  466 538  x000D             For 10000  Total time required for reading and writing in seconds 479 701 x000D     x000D      x000D      x000D  public class DealWithHugeRecordsinFile extends TestCase   x000D   x000D   static final String FILEPATH    C   springbatch  bigfile1 txt txt   x000D   static final String FILEPATH WRITE    C   springbatch  writinghere txt   x000D   static volatile RandomAccessFile fileToWrite  x000D   static volatile RandomAccessFile file  x000D   static volatile String fileContentsIter  x000D   static volatile int position   0  x000D   x000D   public static void main String   args  throws IOException  InterruptedException   x000D    long currentTimeMillis   System currentTimeMillis    x000D   x000D    try   x000D     fileToWrite   new RandomAccessFile FILEPATH WRITE   rw     for random write independent of thread obstacles  x000D     file   new RandomAccessFile FILEPATH   r     for random read independent of thread obstacles  x000D     seriouslyReadProcessAndWriteAsynch    x000D   x000D      catch  IOException e    x000D        TODO Auto-generated catch block x000D     e printStackTrace    x000D      x000D    Thread currentThread   Thread currentThread    x000D    System out println currentThread getName     x000D    long currentTimeMillis2   System currentTimeMillis    x000D    double time seconds    currentTimeMillis2 - currentTimeMillis    1000 0  x000D    System out println  Total time required for reading the text in seconds     time seconds   x000D   x000D     x000D   x000D       x000D       throws IOException x000D      Something  asynchronously serious x000D       x000D   public static void seriouslyReadProcessAndWriteAsynch   throws IOException   x000D    ExecutorService executor   Executors newFixedThreadPool 10    pls see for explanation in comments section of the class x000D    while  true    x000D     String readLine   file readLine    x000D     if  readLine    null    x000D      break  x000D       x000D     Runnable genuineWorker   new Runnable     x000D       Override x000D      public void run     x000D          do hard processing here in this thread i have consumed x000D          some time and ignore some exception in write method  x000D       writeToFile FILEPATH WRITE  readLine   x000D          System out println        x000D          Thread currentThread   getName     x000D   x000D        x000D        x000D     executor execute genuineWorker   x000D      x000D    executor shutdown    x000D    while   executor isTerminated      x000D      x000D    System out println  Finished all threads    x000D    file close    x000D    fileToWrite close    x000D     x000D   x000D       x000D       param filePath x000D       param data x000D       param position x000D       x000D   private static void writeToFile String filePath  String data    x000D    try   x000D        fileToWrite seek position   x000D     data     n    data  x000D     if   data contains  Randomization      x000D      return  x000D       x000D     System out println  Let us do something time consuming to make this thread busy   position               data   x000D     System out println  Lets consume through this loop    x000D     int i 1000  x000D     while i gt 0   x000D      x000D      i--  x000D       x000D     fileToWrite write data getBytes     x000D     throw new Exception    x000D      catch  Exception exception    x000D     System out println  exception was thrown but still we are able to proceeed further  x000D            n This can be used for marking failure of the records    x000D       exception printStackTrace    x000D   x000D      x000D   x000D     x000D    x000D   x000D   x000D

[java] Read large files in Java

Examples related to java

Examples related to memory-management

Examples related to file