[batch-file] Split text file into smaller multiple text file using command line

I have multiple text file with about 100,000 lines and I want to split them into smaller text files of 5000 lines each.

I used:

split -l 5000 filename.txt

That creates files:

xaa
xab
aac
xad
xbe
aaf

files with no extensions. I just want to call them something like:

file01.txt
file02.txt
file03.txt
file04.txt

or if that is not possible, i just want them to have the ".txt" extension.

This question is related to batch-file split command cygwin text-files

The answer is


Here's an example in C# (cause that's what I was searching for). I needed to split a 23 GB csv-file with around 175 million lines to be able to look at the files. I split it into files of one million rows each. This code did it in about 5 minutes on my machine:

var list = new List<string>();
var fileSuffix = 0;

using (var file = File.OpenRead(@"D:\Temp\file.csv"))
using (var reader = new StreamReader(file))
{
    while (!reader.EndOfStream)
    {
        list.Add(reader.ReadLine());

        if (list.Count >= 1000000)
        {
            File.WriteAllLines(@"D:\Temp\split" + (++fileSuffix) + ".csv", list);
            list = new List<string>();
        }
    }
}

File.WriteAllLines(@"D:\Temp\split" + (++fileSuffix) + ".csv", list);

@ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET /a fcount=100
SET /a llimit=5000
SET /a lcount=%llimit%
FOR /f "usebackqdelims=" %%a IN ("%sourcedir%\q25249516.txt") DO (
 CALL :select
 FOR /f "tokens=1*delims==" %%b IN ('set dfile') DO IF /i "%%b"=="dfile" >>"%%c" ECHO(%%a
)
GOTO :EOF
:select
SET /a lcount+=1
IF %lcount% lss %llimit% GOTO :EOF
SET /a lcount=0
SET /a fcount+=1
SET "dfile=%sourcedir%\file%fcount:~-2%.txt"
GOTO :EOF

Here's a native windows batch that should accomplish the task.

Now I'll not say that it'll be fast (less than 2 minutes for each 5Kline output file) or that it will be immune to batch character-sensitivites. Really depends on the characteristics of your target data.

I used a file named q25249516.txt containing 100Klines of data for my testing.


Revised quicker version

REM

@ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET /a fcount=199
SET /a llimit=5000
SET /a lcount=%llimit%
FOR /f "usebackqdelims=" %%a IN ("%sourcedir%\q25249516.txt") DO (
 CALL :select
 >>"%sourcedir%\file$$.txt" ECHO(%%a
)
SET /a lcount=%llimit%
:select
SET /a lcount+=1
IF %lcount% lss %llimit% GOTO :EOF
SET /a lcount=0
SET /a fcount+=1
MOVE /y "%sourcedir%\file$$.txt" "%sourcedir%\file%fcount:~-2%.txt" >NUL 2>nul
GOTO :EOF

Note that I used llimit of 50000 for testing. Will overwrite the early file numbers if llimit*100 is gearter than the number of lines in the file (cure by setting fcount to 1999 and use ~3 in place of ~2 in file-renaming line.)


I know the question has been asked a long time ago, but I am surprised that nobody has given the most straightforward unix answer:

split -l 5000 -d --additional-suffix=.txt $FileName file
  • -l 5000: split file into files of 5,000 lines each.
  • -d: numerical suffix. This will make the suffix go from 00 to 99 by default instead of aa to zz.
  • --additional-suffix: lets you specify the suffix, here the extension
  • $FileName: name of the file to be split.
  • file: prefix to add to the resulting files.

As always, check out man split for more details.

For Mac, the default version of split is apparently dumbed down. You can install the GNU version using the following command. (see this question for more GNU utils)

brew install coreutils

and then you can run the above command by replacing split with gsplit. Check out man gsplit for details.


You can maybe do something like this with awk

awk '{outfile=sprintf("file%02d.txt",NR/5000+1);print > outfile}' yourfile

Basically, it calculates the name of the output file by taking the record number (NR) and dividing it by 5000, adding 1, taking the integer of that and zero-padding to 2 places.

By default, awk prints the entire input record when you don't specify anything else. So, print > outfile writes the entire input record to the output file.

As you are running on Windows, you can't use single quotes because it doesn't like that. I think you have to put the script in a file and then tell awkto use the file, something like this:

awk -f script.awk yourfile

and script.awk will contain the script like this:

{outfile=sprintf("file%02d.txt",NR/5000+1);print > outfile}

Or, it may work if you do this:

awk "{outfile=sprintf(\"file%02d.txt\",NR/5000+1);print > outfile}" yourfile

This "File Splitter" Windows command line program works nicely: https://github.com/dubasdey/File-Splitter

It's open source, simple, documented, proven, and worked for me.

Example:

fsplit -split 50 mb mylargefile.txt

Syntax looks like:

$ split [OPTION] [INPUT [PREFIX]] 

where prefix is PREFIXaa, PREFIXab, ...

Just use proper one and youre done or just use mv for renameing. I think $ mv * *.txt should work but test it first on smaller scale.

:)


My requirement was a bit different. I often work with Comma Delimited and Tab Delimited ASCII files where a single line is a single record of data. And they're really big, so I need to split them into manageable parts (whilst preserving the header row).

So, I reverted back to my classic VBScript method and bashed together a small .vbs script that can be run on any Windows computer (it gets automatically executed by the WScript.exe script host engine on Window).

The benefit of this method is that it uses Text Streams, so the underlying data isn't loaded into memory (or, at least, not all at once). The result is that it's exceptionally fast and it doesn't really need much memory to run. The test file I just split using this script on my i7 was about 1 GB in file size, had about 12 million lines of test and made 25 part files (each with about 500k lines each) – the processing took about 2 minutes and it didn’t go over 3 MB memory used at any point.

The caveat here is that it relies on the text file having "lines" (meaning each record is delimited with a CRLF) as the Text Stream object uses the "ReadLine" function to process a single line at a time. But hey, if you're working with TSV or CSV files, it's perfect.

Option Explicit

Private Const INPUT_TEXT_FILE = "c:\bigtextfile.txt"  'The full path to the big file
Private Const REPEAT_HEADER_ROW = True                'Set to True to duplicate the header row in each part file
Private Const LINES_PER_PART = 500000                 'The number of lines per part file

Dim oFileSystem, oInputFile, oOutputFile, iOutputFile, iLineCounter, sHeaderLine, sLine, sFileExt, sStart

sStart = Now()

sFileExt = Right(INPUT_TEXT_FILE,Len(INPUT_TEXT_FILE)-InstrRev(INPUT_TEXT_FILE,".")+1)
iLineCounter = 0
iOutputFile = 1

Set oFileSystem = CreateObject("Scripting.FileSystemObject")
Set oInputFile = oFileSystem.OpenTextFile(INPUT_TEXT_FILE, 1, False)
Set oOutputFile = oFileSystem.OpenTextFile(Replace(INPUT_TEXT_FILE, sFileExt, "_" & iOutputFile & sFileExt), 2, True)

If REPEAT_HEADER_ROW Then
    iLineCounter = 1
    sHeaderLine = oInputFile.ReadLine()
    Call oOutputFile.WriteLine(sHeaderLine)
End If

Do While Not oInputFile.AtEndOfStream
    sLine = oInputFile.ReadLine()
    Call oOutputFile.WriteLine(sLine)
    iLineCounter = iLineCounter + 1
    If iLineCounter Mod LINES_PER_PART = 0 Then
        iOutputFile = iOutputFile + 1
        Call oOutputFile.Close()
        Set oOutputFile = oFileSystem.OpenTextFile(Replace(INPUT_TEXT_FILE, sFileExt, "_" & iOutputFile & sFileExt), 2, True)
        If REPEAT_HEADER_ROW Then
            Call oOutputFile.WriteLine(sHeaderLine)
        End If
    End If
Loop

Call oInputFile.Close()
Call oOutputFile.Close()
Set oFileSystem = Nothing

Call MsgBox("Done" & vbCrLf & "Lines Processed:" & iLineCounter & vbCrLf & "Part Files: " & iOutputFile & vbCrLf & "Start Time: " & sStart & vbCrLf & "Finish Time: " & Now())

I have created a simple program for this and your question helped me complete the solution... I added one more feature and few configurations. In case you want to add a specific character/ string after every few lines (configurable). Please go through the notes. I have added the code files : https://github.com/mohitsharma779/FileSplit


here is one in c# that doesn't run out of memory when splitting into large chunks! I needed to split 95M file into 10M x line files.

var fileSuffix = 0;
int lines = 0;
Stream fstream = File.OpenWrite($"{filename}.{(++fileSuffix)}");
StreamWriter sw = new StreamWriter(fstream);

using (var file = File.OpenRead(filename))
using (var reader = new StreamReader(file))
{
    while (!reader.EndOfStream)
    {
        sw.WriteLine(reader.ReadLine());
        lines++;

        if (lines >= 10000000)
        {
              sw.Close();
              fstream.Close();
              lines = 0;
              fstream = File.OpenWrite($"{filename}.{(++fileSuffix)}");
              sw = new StreamWriter(fstream);
        }
    }
}

sw.Close();
fstream.Close();

Examples related to batch-file

'ls' is not recognized as an internal or external command, operable program or batch file '' is not recognized as an internal or external command, operable program or batch file XCOPY: Overwrite all without prompt in BATCH Can´t run .bat file under windows 10 Execute a batch file on a remote PC using a batch file on local PC Windows batch - concatenate multiple text files into one How do I create a shortcut via command-line in Windows? Getting Error:JRE_HOME variable is not defined correctly when trying to run startup.bat of Apache-Tomcat Curl not recognized as an internal or external command, operable program or batch file Best way to script remote SSH commands in Batch (Windows)

Examples related to split

Parameter "stratify" from method "train_test_split" (scikit Learn) Pandas split DataFrame by column value How to split large text file in windows? Attribute Error: 'list' object has no attribute 'split' Split function in oracle to comma separated values with automatic sequence How would I get everything before a : in a string Python Split String by delimiter position using oracle SQL JavaScript split String with white space Split a String into an array in Swift? Split pandas dataframe in two if it has more than 10 rows

Examples related to command

'ls' is not recognized as an internal or external command, operable program or batch file Command to run a .bat file how to run python files in windows command prompt? Run a command shell in jenkins How to recover the deleted files using "rm -R" command in linux server? Split text file into smaller multiple text file using command line ansible : how to pass multiple commands Jmeter - Run .jmx file through command line and get the summary report in a excel cocoapods - 'pod install' takes forever Daemon not running. Starting it now on port 5037

Examples related to cygwin

How to install MinGW-w64 and MSYS2? Split text file into smaller multiple text file using command line fatal: early EOF fatal: index-pack failed Git Extensions: Win32 error 487: Couldn't reserve space for cygwin's heap, Win32 error 0 How to cd into a directory with space in the name? Git push hangs when pushing to Github? Running a shell script through Cygwin on Windows Running Git through Cygwin from Windows .ssh directory not being created Cygwin - Makefile-error: recipe for target `main.o' failed

Examples related to text-files

How to edit a text file in my terminal Reading specific columns from a text file in python Read a local text file using Javascript Split text file into smaller multiple text file using command line How do I read a text file of about 2 GB? Java - How Can I Write My ArrayList to a file, and Read (load) that file to the original ArrayList? Copying from one text file to another using Python Getting all file names from a folder using C# Reading From A Text File - Batch How to import data from text file to mysql database