Fastest way to write huge data in text file Java

Question

I have to write huge data in text csv  file  I used BufferedWriter to write the data and it took around 40 secs to write 174 mb of data  Is this the fastest speed java can offer   bufferedWriter   new BufferedWriter   new FileWriter    fileName csv         Note  These 40 secs include the time of iterating and fetching the records from resultset as well       174 mb is for 400000 rows in resultset

User · Answer

Your transfer speed is likely not to be limited by Java. Instead I would suspect (in no particular order)

the speed of transfer from the database
the speed of transfer to the disk

If you read the complete dataset and then write it out to disk, then that will take longer, since the JVM will have to allocate memory, and the db rea/disk write will happen sequentially. Instead I would write out to the buffered writer for every read that you make from the db, and so the operation will be closer to a concurrent one (I don't know if you're doing that or not)

User · Answer

x000D   x000D  package all is well  x000D  import java io IOException  x000D  import java io RandomAccessFile  x000D  import java util concurrent ExecutorService  x000D  import java util concurrent Executors  x000D  import junit framework TestCase  x000D   x000D      x000D      author Naresh Bhabat x000D      x000D  Following  implementation helps to deal with extra large files in java  x000D  This program is tested for dealing with 2GB input file  x000D  There are some points where extra logic can be added in future  x000D   x000D   x000D  Pleasenote  if we want to deal with binary input file  then instead of reading line we need to read bytes from read file object  x000D   x000D   x000D   x000D  It uses random access file which is almost like streaming API  x000D   x000D   x000D                                              x000D  Notes regarding executor framework and its readings  x000D  Please note  ExecutorService executor   Executors newFixedThreadPool 10   x000D   x000D          for 10 threads Total time required for reading and writing the text in x000D              seconds 349 317 x000D      x000D             For 100 Total time required for reading the text and writing     seconds 464 042 x000D      x000D             For 1000   Total time required for reading and writing text  466 538  x000D             For 10000  Total time required for reading and writing in seconds 479 701 x000D     x000D      x000D      x000D  public class DealWithHugeRecordsinFile extends TestCase   x000D   x000D   static final String FILEPATH    C   springbatch  bigfile1 txt txt   x000D   static final String FILEPATH WRITE    C   springbatch  writinghere txt   x000D   static volatile RandomAccessFile fileToWrite  x000D   static volatile RandomAccessFile file  x000D   static volatile String fileContentsIter  x000D   static volatile int position   0  x000D   x000D   public static void main String   args  throws IOException  InterruptedException   x000D    long currentTimeMillis   System currentTimeMillis    x000D   x000D    try   x000D     fileToWrite   new RandomAccessFile FILEPATH WRITE   rw     for random write independent of thread obstacles  x000D     file   new RandomAccessFile FILEPATH   r     for random read independent of thread obstacles  x000D     seriouslyReadProcessAndWriteAsynch    x000D   x000D      catch  IOException e    x000D        TODO Auto-generated catch block x000D     e printStackTrace    x000D      x000D    Thread currentThread   Thread currentThread    x000D    System out println currentThread getName     x000D    long currentTimeMillis2   System currentTimeMillis    x000D    double time seconds    currentTimeMillis2 - currentTimeMillis    1000 0  x000D    System out println  Total time required for reading the text in seconds     time seconds   x000D   x000D     x000D   x000D       x000D       throws IOException x000D      Something  asynchronously serious x000D       x000D   public static void seriouslyReadProcessAndWriteAsynch   throws IOException   x000D    ExecutorService executor   Executors newFixedThreadPool 10    pls see for explanation in comments section of the class x000D    while  true    x000D     String readLine   file readLine    x000D     if  readLine    null    x000D      break  x000D       x000D     Runnable genuineWorker   new Runnable     x000D       Override x000D      public void run     x000D          do hard processing here in this thread i have consumed x000D          some time and eat some exception in write method  x000D       writeToFile FILEPATH WRITE  readLine   x000D          System out println        x000D          Thread currentThread   getName     x000D   x000D        x000D        x000D     executor execute genuineWorker   x000D      x000D    executor shutdown    x000D    while   executor isTerminated      x000D      x000D    System out println  Finished all threads    x000D    file close    x000D    fileToWrite close    x000D     x000D   x000D       x000D       param filePath x000D       param data x000D       param position x000D       x000D   private static void writeToFile String filePath  String data    x000D    try   x000D        fileToWrite seek position   x000D     data     n    data  x000D     if   data contains  Randomization      x000D      return  x000D       x000D     System out println  Let us do something time consuming to make this thread busy   position               data   x000D     System out println  Lets consume through this loop    x000D     int i 1000  x000D     while i gt 0   x000D      x000D      i--  x000D       x000D     fileToWrite write data getBytes     x000D     throw new Exception    x000D      catch  Exception exception    x000D     System out println  exception was thrown but still we are able to proceeed further  x000D            n This can be used for marking failure of the records    x000D       exception printStackTrace    x000D   x000D      x000D   x000D     x000D    x000D   x000D   x000D

User · Answer

try memory mapped files  takes 300 m s to write 174MB in my m c  core 2 duo  2 5GB RAM     byte   buffer    Help I am trapped in a fortune cookie factory n  getBytes    int number of lines   400000   FileChannel rwChannel   new RandomAccessFile  textfile txt    rw   getChannel    ByteBuffer wrBuf   rwChannel map FileChannel MapMode READ WRITE  0  buffer length   number of lines   for  int i   0  i  lt  number of lines  i          wrBuf put buffer     rwChannel close

User · Answer

For those who want to improve the time for retrieval of records and dump into the file  i e no processing on records   instead of putting them into an ArrayList  append those records into a StringBuffer  Apply toSring   function to get a single String and write it into the file at once    For me  the retrieval time reduced from 22 seconds to 17 seconds

User · Answer

For these bulky reads from DB you may want to tune your Statement s fetch size  It might save a lot of roundtrips to DB   http   download oracle com javase 1 5 0 docs api java sql Statement html setFetchSize 28int 29

User · Answer

Only for the sake of statistics    The machine is old Dell with new SSD  CPU  Intel Pentium D 2 8 Ghz  SSD  Patriot Inferno 120GB SSD  4000000  records  175 47607421875 MB  Iteration 0 Writing raw    3 547 seconds Writing buffered  buffer size  8192     2 625 seconds Writing buffered  buffer size  1048576     2 203 seconds Writing buffered  buffer size  4194304     2 312 seconds  Iteration 1 Writing raw    2 922 seconds Writing buffered  buffer size  8192     2 406 seconds Writing buffered  buffer size  1048576     2 015 seconds Writing buffered  buffer size  4194304     2 282 seconds  Iteration 2 Writing raw    2 828 seconds Writing buffered  buffer size  8192     2 109 seconds Writing buffered  buffer size  1048576     2 078 seconds Writing buffered  buffer size  4194304     2 015 seconds  Iteration 3 Writing raw    3 187 seconds Writing buffered  buffer size  8192     2 109 seconds Writing buffered  buffer size  1048576     2 094 seconds Writing buffered  buffer size  4194304     2 031 seconds  Iteration 4 Writing raw    3 093 seconds Writing buffered  buffer size  8192     2 141 seconds Writing buffered  buffer size  1048576     2 063 seconds Writing buffered  buffer size  4194304     2 016 seconds   As we can see the raw method is slower the buffered

User · Answer

You might try removing the BufferedWriter and just using the FileWriter directly  On a modern system there s a good chance you re just writing to the drive s cache memory anyway  It takes me in the range of 4-5 seconds to write 175MB  4 million strings  -- this is on a dual-core 2 4GHz Dell running Windows XP with an 80GB  7200-RPM Hitachi disk  Can you isolate how much of the time is record retrieval and how much is file writing  import java io BufferedWriter  import java io File  import java io FileWriter  import java io IOException  import java io Writer  import java util ArrayList  import java util List   public class FileWritingPerfTest         private static final int ITERATIONS   5  private static final double MEG    Math pow 1024  2    private static final int RECORD COUNT   4000000  private static final String RECORD    quot Help I am trapped in a fortune cookie factory n quot   private static final int RECSIZE   RECORD getBytes   length   public static void main String   args  throws Exception       List lt String gt  records   new ArrayList lt String gt  RECORD COUNT       int size   0      for  int i   0  i  lt  RECORD COUNT  i              records add RECORD           size    RECSIZE            System out println records size      quot   records  quot        System out println size   MEG    quot  MB quot             for  int i   0  i  lt  ITERATIONS  i              System out println  quot  nIteration  quot    i                    writeRaw records           writeBuffered records  8192           writeBuffered records   int  MEG           writeBuffered records  4    int  MEG            private static void writeRaw List lt String gt  records  throws IOException       File file   File createTempFile  quot foo quot    quot  txt quot        try           FileWriter writer   new FileWriter file           System out print  quot Writing raw     quot            write records  writer         finally              comment this out if you want to inspect the files afterward         file delete             private static void writeBuffered List lt String gt  records  int bufSize  throws IOException       File file   File createTempFile  quot foo quot    quot  txt quot        try           FileWriter writer   new FileWriter file           BufferedWriter bufferedWriter   new BufferedWriter writer  bufSize                System out print  quot Writing buffered  buffer size   quot    bufSize    quot       quot            write records  bufferedWriter         finally              comment this out if you want to inspect the files afterward         file delete             private static void write List lt String gt  records  Writer writer  throws IOException       long start   System currentTimeMillis        for  String record  records            writer write record                writer flush       close   should take care of this     writer close         long end   System currentTimeMillis        System out println  end - start    1000f    quot  seconds quot

[java] Fastest way to write huge data in text file Java

Examples related to java

Examples related to file

Examples related to resultset