Split text file into smaller multiple text file using command line

Question

I have multiple text file with about 100 000 lines and I want to split them into smaller text files of 5000 lines each    I used   split -l 5000 filename txt   That creates files   xaa xab aac xad xbe aaf   files with no extensions  I just want to call them something like   file01 txt file02 txt file03 txt file04 txt   or if that is not possible  i just want them to have the   txt  extension

User · Answer

I know the question has been asked a long time ago, but I am surprised that nobody has given the most straightforward unix answer:

split -l 5000 -d --additional-suffix=.txt $FileName file

-l 5000: split file into files of 5,000 lines each.
-d: numerical suffix. This will make the suffix go from 00 to 99 by default instead of aa to zz.
--additional-suffix: lets you specify the suffix, here the extension
$FileName: name of the file to be split.
file: prefix to add to the resulting files.

As always, check out man split for more details.

For Mac, the default version of split is apparently dumbed down. You can install the GNU version using the following command. (see this question for more GNU utils)

brew install coreutils

and then you can run the above command by replacing split with gsplit. Check out man gsplit for details.

User · Answer

ECHO OFF SETLOCAL SET  sourcedir U  sourcedir  SET  a fcount 100 SET  a llimit 5000 SET  a lcount  llimit  FOR  f  usebackqdelims     a IN    sourcedir  q25249516 txt   DO    CALL  select  FOR  f  tokens 1 delims      b IN   set dfile   DO IF  i    b    dfile   gt  gt    c  ECHO   a   GOTO  EOF  select SET  a lcount  1 IF  lcount  lss  llimit  GOTO  EOF SET  a lcount 0 SET  a fcount  1 SET  dfile  sourcedir  file fcount  -2  txt  GOTO  EOF   Here s a native windows batch that should accomplish the task   Now I ll not say that it ll be fast  less than 2 minutes for each 5Kline output file  or that it will be immune to batch character-sensitivites  Really depends on the characteristics of your target data   I used a file named q25249516 txt containing 100Klines of data for my testing     Revised quicker version  REM    ECHO OFF SETLOCAL SET  sourcedir U  sourcedir  SET  a fcount 199 SET  a llimit 5000 SET  a lcount  llimit  FOR  f  usebackqdelims     a IN    sourcedir  q25249516 txt   DO    CALL  select   gt  gt   sourcedir  file   txt  ECHO   a   SET  a lcount  llimit   select SET  a lcount  1 IF  lcount  lss  llimit  GOTO  EOF SET  a lcount 0 SET  a fcount  1 MOVE  y   sourcedir  file   txt    sourcedir  file fcount  -2  txt   gt NUL 2 gt nul GOTO  EOF   Note that I used llimit of 50000 for testing  Will overwrite the early file numbers if llimit 100 is gearter than the number of lines in the file  cure by setting fcount to 1999 and use  3 in place of  2 in file-renaming line

User · Answer

Syntax looks like     split  OPTION   INPUT  PREFIX      where prefix is  PREFIXaa  PREFIXab        Just use proper one and youre done or just use mv for renameing  I think    mv     txt should work but test it first on smaller scale

User · Answer

Here s an example in C   cause that s what I was searching for   I needed to split a 23 GB csv-file with around 175 million lines to be able to look at the files  I split it into files of one million rows each  This code did it in about 5 minutes on my machine   var list   new List lt string gt     var fileSuffix   0   using  var file   File OpenRead   D  Temp file csv    using  var reader   new StreamReader file         while   reader EndOfStream                list Add reader ReadLine              if  list Count  gt   1000000                        File WriteAllLines   D  Temp split       fileSuffix      csv   list               list   new List lt string gt                        File WriteAllLines   D  Temp split       fileSuffix      csv   list

User · Answer

I have created a simple program for this and your question helped me complete the solution     I added one more feature and few configurations  In case you want to add a specific character  string after every few lines  configurable   Please go through the notes  I have added the code files   https   github com mohitsharma779 FileSplit

User · Answer

here is one in c  that doesn t run out of memory when splitting into large chunks  I needed to split 95M file into 10M x line files   var fileSuffix   0  int lines   0  Stream fstream   File OpenWrite    filename      fileSuffix      StreamWriter sw   new StreamWriter fstream    using  var file   File OpenRead filename   using  var reader   new StreamReader file         while   reader EndOfStream                sw WriteLine reader ReadLine             lines             if  lines  gt   10000000                          sw Close                  fstream Close                  lines   0                fstream   File OpenWrite    filename      fileSuffix                    sw   new StreamWriter fstream                      sw Close    fstream Close

User · Answer

This  File Splitter  Windows command line program works nicely  https   github com dubasdey File-Splitter  It s open source  simple  documented  proven  and worked for me   Example   fsplit -split 50 mb mylargefile txt

User · Answer

You can maybe do something like this with awk  awk   outfile sprintf  file 02d txt  NR 5000 1  print  gt  outfile   yourfile   Basically  it calculates the name of the output file by taking the record number  NR  and dividing it by 5000  adding 1  taking the integer of that and zero-padding to 2 places   By default  awk prints the entire input record when you don t specify anything else  So  print  gt  outfile writes the entire input record to the output file   As you are running on Windows  you can t use single quotes because it doesn t like that  I think you have to put the script in a file and then tell awkto use the file  something like this   awk -f script awk yourfile   and script awk will contain the script like this    outfile sprintf  file 02d txt  NR 5000 1  print  gt  outfile    Or  it may work if you do this   awk   outfile sprintf   file 02d txt   NR 5000 1  print  gt  outfile   yourfile

User · Answer

My requirement was a bit different  I often work with Comma Delimited and Tab Delimited ASCII files where a single line is a single record of data  And they re really big  so I need to split them into manageable parts  whilst preserving the header row    So  I reverted back to my classic VBScript method and bashed together a small  vbs script that can be run on any Windows computer  it gets automatically executed by the WScript exe script host engine on Window    The benefit of this method is that it uses Text Streams  so the underlying data isn t loaded into memory  or  at least  not all at once   The result is that it s exceptionally fast and it doesn t really need much memory to run  The test file I just split using this script on my i7 was about 1 GB in file size  had about 12 million lines of test and made 25 part files  each with about 500k lines each      the processing took about 2 minutes and it didn   t go over 3 MB memory used at any point   The caveat here is that it relies on the text file having  lines   meaning each record is delimited with a CRLF  as the Text Stream object uses the  ReadLine  function to process a single line at a time  But hey  if you re working with TSV or CSV files  it s perfect   Option Explicit  Private Const INPUT TEXT FILE    c  bigtextfile txt    The full path to the big file Private Const REPEAT HEADER ROW   True                 Set to True to duplicate the header row in each part file Private Const LINES PER PART   500000                  The number of lines per part file  Dim oFileSystem  oInputFile  oOutputFile  iOutputFile  iLineCounter  sHeaderLine  sLine  sFileExt  sStart  sStart   Now    sFileExt   Right INPUT TEXT FILE Len INPUT TEXT FILE -InstrRev INPUT TEXT FILE      1  iLineCounter   0 iOutputFile   1  Set oFileSystem   CreateObject  Scripting FileSystemObject   Set oInputFile   oFileSystem OpenTextFile INPUT TEXT FILE  1  False  Set oOutputFile   oFileSystem OpenTextFile Replace INPUT TEXT FILE  sFileExt       amp  iOutputFile  amp  sFileExt   2  True   If REPEAT HEADER ROW Then     iLineCounter   1     sHeaderLine   oInputFile ReadLine       Call oOutputFile WriteLine sHeaderLine  End If  Do While Not oInputFile AtEndOfStream     sLine   oInputFile ReadLine       Call oOutputFile WriteLine sLine      iLineCounter   iLineCounter   1     If iLineCounter Mod LINES PER PART   0 Then         iOutputFile   iOutputFile   1         Call oOutputFile Close           Set oOutputFile   oFileSystem OpenTextFile Replace INPUT TEXT FILE  sFileExt       amp  iOutputFile  amp  sFileExt   2  True          If REPEAT HEADER ROW Then             Call oOutputFile WriteLine sHeaderLine          End If     End If Loop  Call oInputFile Close   Call oOutputFile Close   Set oFileSystem   Nothing  Call MsgBox  Done   amp  vbCrLf  amp   Lines Processed    amp  iLineCounter  amp  vbCrLf  amp   Part Files     amp  iOutputFile  amp  vbCrLf  amp   Start Time     amp  sStart  amp  vbCrLf  amp   Finish Time     amp  Now

[batch-file] Split text file into smaller multiple text file using command line

Examples related to batch-file

Examples related to split

Examples related to command

Examples related to cygwin

Examples related to text-files