[windows] Text editor to open big (giant, huge, large) text files

Tips and tricks

less

Why are you using editors to just look at a (large) file?

Under *nix or Cygwin, just use less. (There is a famous saying – "less is more, more or less" – because "less" replaced the earlier Unix command "more", with the addition that you could scroll back up.) Searching and navigating under less is very similar to Vim, but there is no swap file and little RAM used.

There is a Win32 port of GNU less. See the "less" section of the answer above.

Perl

Perl is good for quick scripts, and its .. (range flip-flop) operator makes for a nice selection mechanism to limit the crud you have to wade through.

For example:

$ perl -n -e 'print if ( 1000000 .. 2000000)' humongo.txt | less

This will extract everything from line 1 million to line 2 million, and allow you to sift the output manually in less.

Another example:

$ perl -n -e 'print if ( /regex one/ .. /regex two/)' humongo.txt | less

This starts printing when the "regular expression one" finds something, and stops when the "regular expression two" find the end of an interesting block. It may find multiple blocks. Sift the output...

logparser

This is another useful tool you can use. To quote the Wikipedia article:

logparser is a flexible command line utility that was initially written by Gabriele Giuseppini, a Microsoft employee, to automate tests for IIS logging. It was intended for use with the Windows operating system, and was included with the IIS 6.0 Resource Kit Tools. The default behavior of logparser works like a "data processing pipeline", by taking an SQL expression on the command line, and outputting the lines containing matches for the SQL expression.

Microsoft describes Logparser as a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows operating system such as the Event Log, the Registry, the file system, and Active Directory. The results of the input query can be custom-formatted in text based output, or they can be persisted to more specialty targets like SQL, SYSLOG, or a chart.

Example usage:

C:\>logparser.exe -i:textline -o:tsv "select Index, Text from 'c:\path\to\file.log' where line > 1000 and line < 2000"
C:\>logparser.exe -i:textline -o:tsv "select Index, Text from 'c:\path\to\file.log' where line like '%pattern%'"

The relativity of sizes

100 MB isn't too big. 3 GB is getting kind of big. I used to work at a print & mail facility that created about 2% of U.S. first class mail. One of the systems for which I was the tech lead accounted for about 15+% of the pieces of mail. We had some big files to debug here and there.

And more...

Feel free to add more tools and information here. This answer is community wiki for a reason! We all need more advice on dealing with large amounts of data...

Examples related to windows

"Permission Denied" trying to run Python on Windows 10 A fatal error occurred while creating a TLS client credential. The internal error state is 10013 How to install OpenJDK 11 on Windows? I can't install pyaudio on Windows? How to solve "error: Microsoft Visual C++ 14.0 is required."? git clone: Authentication failed for <URL> How to avoid the "Windows Defender SmartScreen prevented an unrecognized app from starting warning" XCOPY: Overwrite all without prompt in BATCH Laravel 5 show ErrorException file_put_contents failed to open stream: No such file or directory how to open Jupyter notebook in chrome on windows Tensorflow import error: No module named 'tensorflow'

Examples related to xml

strange error in my Animation Drawable How do I POST XML data to a webservice with Postman? PHP XML Extension: Not installed How to add a Hint in spinner in XML Generating Request/Response XML from a WSDL Manifest Merger failed with multiple errors in Android Studio How to set menu to Toolbar in Android How to add colored border on cardview? Android: ScrollView vs NestedScrollView WARNING: Exception encountered during context initialization - cancelling refresh attempt

Examples related to editor

Select all occurrences of selected word in VSCode Change the Theme in Jupyter Notebook? How to view Plugin Manager in Notepad++ Set language for syntax highlighting in Visual Studio Code Copy text from nano editor to shell How do I duplicate a line or selection within Visual Studio Code? How to set editor theme in IntelliJ Idea How to change background color in the Notepad++ text editor? What is the difference between Sublime text and Github's Atom What are the advantages of Sublime Text over Notepad++ and vice-versa?

Examples related to text-editor

Find duplicates and delete all in notepad++ How to run vi on docker container? How to run a program in Atom Editor? How to call VS Code Editor from terminal / command line What is the difference between Sublime text and Github's Atom Multiple select in Visual Studio? change cursor from block or rectangle to line? Javascript button to insert a big black dot (•) into a html textarea Vim multiline editing like in sublimetext? Edit a text file on the console using Powershell

Examples related to large-files

How can I import a large (14 GB) MySQL dump file into a new MySQL database? Read and parse a Json File in C# How to find the largest file in a directory and its subdirectories? Reading large text files with streams in C# Working with huge files in VIM What is the fastest way to create a checksum for large files in C# How to read large text file on windows? Managing large binary files with Git Number of lines in a file in Java Text editor to open big (giant, huge, large) text files