[unix] Move top 1000 lines from text file to a new file using Unix shell commands

I wish to copy the top 1000 lines in a text file containing more than 50 million entries, to another new file, and also delete these lines from the original file.

Is there some way to do the same with a single shell command in Unix?

This question is related to unix shell copy

The answer is


Perl approach:

perl -ne 'if($i<1000) { print; } else { print STDERR;}; $i++;' in 1> in.new 2> out && mv in.new in

head -1000 file.txt > first100lines.txt
tail --lines=+1001 file.txt > restoffile.txt

Using pipe:

cat en-tl.100.en | head -10

Out of curiosity, I found a box with a GNU version of sed (v4.1.5) and tested the (uncached) performance of two approaches suggested so far, using an 11M line text file:

$ wc -l input
11771722 input

$ time head -1000 input > output; time tail -n +1000 input > input.tmp; time cp input.tmp input; time rm input.tmp

real    0m1.165s
user    0m0.030s
sys     0m1.130s

real    0m1.256s
user    0m0.062s
sys     0m1.162s

real    0m4.433s
user    0m0.033s
sys     0m1.282s

real    0m6.897s
user    0m0.000s
sys     0m0.159s

$ time head -1000 input > output && time sed -i '1,+999d' input

real    0m0.121s
user    0m0.000s
sys     0m0.121s

real    0m26.944s
user    0m0.227s
sys     0m26.624s

This is the Linux I was working with:

$ uname -a
Linux hostname 2.6.18-128.1.1.el5 #1 SMP Mon Jan 26 13:58:24 EST 2009 x86_64 x86_64 x86_64 GNU/Linux

For this test, at least, it looks like sed is slower than the tail approach (27 sec vs ~14 sec).


This is a one-liner but uses four atomic commands:

head -1000 file.txt > newfile.txt; tail +1000 file.txt > file.txt.tmp; cp file.txt.tmp file.txt; rm file.txt.tmp

Examples related to unix

Docker CE on RHEL - Requires: container-selinux >= 2.9 What does `set -x` do? How to find files modified in last x minutes (find -mmin does not work as expected) sudo: npm: command not found How to sort a file in-place How to read a .properties file which contains keys that have a period character using Shell script gpg decryption fails with no secret key error Loop through a comma-separated shell variable Best way to find os name and version in Unix/Linux platform Resource u'tokenizers/punkt/english.pickle' not found

Examples related to shell

Comparing a variable with a string python not working when redirecting from bash script Get first line of a shell command's output How to run shell script file using nodejs? Run bash command on jenkins pipeline Way to create multiline comments in Bash? How to do multiline shell script in Ansible How to check if a file exists in a shell script How to check if an environment variable exists and get its value? Curl to return http status code along with the response docker entrypoint running bash script gets "permission denied"

Examples related to copy

Copying files to a container with Docker Compose Copy filtered data to another sheet using VBA Copy output of a JavaScript variable to the clipboard Dockerfile copy keep subdirectory structure Using a batch to copy from network drive to C: or D: drive Copying HTML code in Google Chrome's inspect element What is the difference between `sorted(list)` vs `list.sort()`? How to export all data from table to an insertable sql format? scp copy directory to another server with private key auth How to properly -filter multiple strings in a PowerShell copy script