[windows] Batch file to split .csv file

If splitting very large files, the solution I found is an adaptation from this, with PowerShell "embedded" in a batch file. This works fast, as opposed to many other things I tried (I wouldn't know about other options posted here).

The way to use mysplit.bat below is

mysplit.bat <mysize> 'myfile'

Note: The script was intended to use the first argument as the split size. It is currently hardcoded at 100Mb. It should not be difficult to fix this.

Note 2: The filname should be enclosed in single quotes. Other alternatives for quoting apparently do not work.

Note 3: It splits the file at given number of bytes, not at given number of lines. For me this was good enough. Some lines of code could be probably added to complete each chunk read, up to the next CR/LF. This will split in full lines (not with a constant number of them), with no sacrifice in processing time.

Script mysplit.bat:

@REM Using https://stackoverflow.com/questions/19335004/how-to-run-a-powershell-script-from-a-batch-file
@REM and https://stackoverflow.com/questions/1001776/how-can-i-split-a-text-file-using-powershell
@PowerShell  ^
    $upperBound = 100MB;  ^
    $rootName = %2;  ^
    $from = $rootName;  ^
    $fromFile = [io.file]::OpenRead($from);  ^
    $buff = new-object byte[] $upperBound;  ^
    $count = $idx = 0;  ^
    try {  ^
        do {  ^
            'Reading ' + $upperBound;  ^
            $count = $fromFile.Read($buff, 0, $buff.Length);  ^
            if ($count -gt 0) {  ^
                $to = '{0}.{1}' -f ($rootName, $idx);  ^
                $toFile = [io.file]::OpenWrite($to);  ^
                try {  ^
                    'Writing ' + $count + ' to ' + $to;  ^
                    $tofile.Write($buff, 0, $count);  ^
                } finally {  ^
                    $tofile.Close();  ^
                }  ^
            }  ^
            $idx ++;  ^
        } while ($count -gt 0);  ^
    }  ^
    finally {  ^
        $fromFile.Close();  ^
    }  ^
%End PowerShell%

Examples related to windows

"Permission Denied" trying to run Python on Windows 10 A fatal error occurred while creating a TLS client credential. The internal error state is 10013 How to install OpenJDK 11 on Windows? I can't install pyaudio on Windows? How to solve "error: Microsoft Visual C++ 14.0 is required."? git clone: Authentication failed for <URL> How to avoid the "Windows Defender SmartScreen prevented an unrecognized app from starting warning" XCOPY: Overwrite all without prompt in BATCH Laravel 5 show ErrorException file_put_contents failed to open stream: No such file or directory how to open Jupyter notebook in chrome on windows Tensorflow import error: No module named 'tensorflow'

Examples related to batch-file

'ls' is not recognized as an internal or external command, operable program or batch file '' is not recognized as an internal or external command, operable program or batch file XCOPY: Overwrite all without prompt in BATCH CanĀ“t run .bat file under windows 10 Execute a batch file on a remote PC using a batch file on local PC Windows batch - concatenate multiple text files into one How do I create a shortcut via command-line in Windows? Getting Error:JRE_HOME variable is not defined correctly when trying to run startup.bat of Apache-Tomcat Curl not recognized as an internal or external command, operable program or batch file Best way to script remote SSH commands in Batch (Windows)

Examples related to csv

Pandas: ValueError: cannot convert float NaN to integer Export result set on Dbeaver to CSV Convert txt to csv python script How to import an Excel file into SQL Server? "CSV file does not exist" for a filename with embedded quotes Save Dataframe to csv directly to s3 Python Data-frame Object has no Attribute (unicode error) 'unicodeescape' codec can't decode bytes in position 2-3: truncated \UXXXXXXXX escape How to write to a CSV line by line? How to check encoding of a CSV file

Examples related to command-prompt

CMD command to check connected USB devices How do I kill the process currently using a port on localhost in Windows? PowerShell The term is not recognized as cmdlet function script file or operable program open program minimized via command prompt How to connect to SQL Server from command prompt with Windows authentication How to see the proxy settings on windows? How do I type a TAB character in PowerShell? Batch file to split .csv file Aliases in Windows command prompt Change all files and folders permissions of a directory to 644/755