I have a ~23000 line SQL dump containing several databases worth of data. I need to extract a certain section of this file (i.e. the data for a single database) and place it in a new file. I know both the start and end line numbers of the data that I want.
Does anyone know a Unix command (or series of commands) to extract all lines from a file between say line 16224 and 16482 and then redirect them into a new file?
This question is related to
unix
command-line
sed
text-processing
sed -n '16224,16482p' < dump.sql
Just benchmarking 3 solutions given above, that works to me:
Credits on the 3 solutions goes to:
I'm using a huge file I find in my server:
# wc fo2debug.1.log
10421186 19448208 38795491134 fo2debug.1.log
38 Gb in 10.4 million lines.
And yes, I have a logrotate problem. : ))
Getting 256 lines from the beginning of the file.
# time sed -n '1001,1256p;1256q' fo2debug.1.log | wc -l
256
real 0m0,003s
user 0m0,000s
sys 0m0,004s
# time head -1256 fo2debug.1.log | tail -n +1001 | wc -l
256
real 0m0,003s
user 0m0,006s
sys 0m0,000s
# time awk 'NR==1001, NR==1256; NR==1256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,002s
user 0m0,004s
sys 0m0,000s
Awk won. Technical tie in second place between sed and "head+tail".
Getting 256 lines at the end of the first third of the file.
# time sed -n '3473001,3473256p;3473256q' fo2debug.1.log | wc -l
256
real 0m0,265s
user 0m0,242s
sys 0m0,024s
# time head -3473256 fo2debug.1.log | tail -n +3473001 | wc -l
256
real 0m0,308s
user 0m0,313s
sys 0m0,145s
# time awk 'NR==3473001, NR==3473256; NR==3473256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,393s
user 0m0,326s
sys 0m0,068s
Sed won. Followed by "head+tail" and, finally, awk.
Getting 256 lines at the end of the second third of the file.
# time sed -n '6947001,6947256p;6947256q' fo2debug.1.log | wc -l
A256
real 0m0,525s
user 0m0,462s
sys 0m0,064s
# time head -6947256 fo2debug.1.log | tail -n +6947001 | wc -l
256
real 0m0,615s
user 0m0,488s
sys 0m0,423s
# time awk 'NR==6947001, NR==6947256; NR==6947256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,779s
user 0m0,650s
sys 0m0,130s
Same results.
Sed won. Followed by "head+tail" and, finally, awk.
Getting 256 lines near the end of the file.
# time sed -n '10420001,10420256p;10420256q' fo2debug.1.log | wc -l
256
real 1m50,017s
user 0m12,735s
sys 0m22,926s
# time head -10420256 fo2debug.1.log | tail -n +10420001 | wc -l
256
real 1m48,269s
user 0m42,404s
sys 0m51,015s
# time awk 'NR==10420001, NR==10420256; NR==10420256 {exit}' fo2debug.1.log | wc -l
256
real 1m49,106s
user 0m12,322s
sys 0m18,576s
And suddenly, a twist!
"Head+tail" won. Followed by awk and, finally, sed.
(some hours later...)
My analysis above ends up being an example of a basic flaw in doing an analysis.
The flaw is not knowing in depth the resources used for the analysis.
In this case, I used a log file to analyze the performance of a search for a certain number of lines within it.
Using 3 different techniques, searches were made at different points in the file, comparing the performance of the techniques at each point and checking whether the results varied depending on the point in the file where the search was made.
My mistake was to assume that there was a certain homogeneity of content in the log file.
The reality is that long lines appear more frequently at the end of the file.
Thus, the apparent conclusion that longer searches (closer to the end of the file) are better with a given technique, may be biased. In fact, this technique may be better when dealing with longer lines. What remains to be confirmed.
sed -n '16224,16482 p' orig-data-file > new-file
Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. -n suppresses echoing the input as output, which you clearly don't want; the numbers indicate the range of lines to make the following command operate on; the command p prints out the relevant lines.
# print section of file based on line numbers
sed -n '16224 ,16482p' # method 1
sed '16224,16482!d' # method 2
Quite simple using head/tail:
head -16482 in.sql | tail -258 > out.sql
using sed:
sed -n '16224,16482p' in.sql > out.sql
using awk:
awk 'NR>=16224&&NR<=16482' in.sql > out.sql
You could use 'vi' and then the following command:
:16224,16482w!/tmp/some-file
Alternatively:
cat file | head -n 16482 | tail -n 258
EDIT:- Just to add explanation, you use head -n 16482 to display first 16482 lines then use tail -n 258 to get last 258 lines out of the first output.
cat dump.txt | head -16224 | tail -258
should do the trick. The downside of this approach is that you need to do the arithmetic to determine the argument for tail and to account for whether you want the 'between' to include the ending line or not.
I was looking for an answer to this but I had to end up writing my own code which worked. None of the answers above were satisfactory. Consider you have very large file and have certain line numbers that you want to print out but the numbers are not in order. You can do the following:
My relatively large file
for letter in {a..k} ; do echo $letter; done | cat -n > myfile.txt
1 a
2 b
3 c
4 d
5 e
6 f
7 g
8 h
9 i
10 j
11 k
Specific line numbers I want:
shuf -i 1-11 -n 4 > line_numbers_I_want.txt
10
11
4
9
To print these line numbers, do the following.
awk '{system("head myfile.txt -n " $0 " | tail -n 1")}' line_numbers_I_want.txt
What the above does is to head the n line then take the last line using tail
If you want your line numbers in order, sort ( is -n numeric sort) first then get the lines.
cat line_numbers_I_want.txt | sort -n | awk '{system("head myfile.txt -n " $0 " | tail -n 1")}'
4 d
9 i
10 j
11 k
Using ed:
ed -s infile <<<'16224,16482p'
-s suppresses diagnostic output; the actual commands are in a here-string. Specifically, 16224,16482p runs the p (print) command on the desired line address range.
Quite simple using head/tail:
head -16482 in.sql | tail -258 > out.sql
using sed:
sed -n '16224,16482p' in.sql > out.sql
using awk:
awk 'NR>=16224&&NR<=16482' in.sql > out.sql
You could use 'vi' and then the following command:
:16224,16482w!/tmp/some-file
Alternatively:
cat file | head -n 16482 | tail -n 258
EDIT:- Just to add explanation, you use head -n 16482 to display first 16482 lines then use tail -n 258 to get last 258 lines out of the first output.
Quite simple using head/tail:
head -16482 in.sql | tail -258 > out.sql
using sed:
sed -n '16224,16482p' in.sql > out.sql
using awk:
awk 'NR>=16224&&NR<=16482' in.sql > out.sql
perl -ne 'print if 16224..16482' file.txt > new_file.txt
Even we can do this to check at command line:
cat filename|sed 'n1,n2!d' > abc.txt
For Example:
cat foo.pl|sed '100,200!d' > abc.txt
I was about to post the head/tail trick, but actually I'd probably just fire up emacs. ;-)
open the new output file, ctl-y save
Let's me see what's happening.
sed -n '16224,16482 p' orig-data-file > new-file
Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. -n suppresses echoing the input as output, which you clearly don't want; the numbers indicate the range of lines to make the following command operate on; the command p prints out the relevant lines.
I was looking for an answer to this but I had to end up writing my own code which worked. None of the answers above were satisfactory. Consider you have very large file and have certain line numbers that you want to print out but the numbers are not in order. You can do the following:
My relatively large file
for letter in {a..k} ; do echo $letter; done | cat -n > myfile.txt
1 a
2 b
3 c
4 d
5 e
6 f
7 g
8 h
9 i
10 j
11 k
Specific line numbers I want:
shuf -i 1-11 -n 4 > line_numbers_I_want.txt
10
11
4
9
To print these line numbers, do the following.
awk '{system("head myfile.txt -n " $0 " | tail -n 1")}' line_numbers_I_want.txt
What the above does is to head the n line then take the last line using tail
If you want your line numbers in order, sort ( is -n numeric sort) first then get the lines.
cat line_numbers_I_want.txt | sort -n | awk '{system("head myfile.txt -n " $0 " | tail -n 1")}'
4 d
9 i
10 j
11 k
I was about to post the head/tail trick, but actually I'd probably just fire up emacs. ;-)
open the new output file, ctl-y save
Let's me see what's happening.
perl -ne 'print if 16224..16482' file.txt > new_file.txt
# print section of file based on line numbers
sed -n '16224 ,16482p' # method 1
sed '16224,16482!d' # method 2
perl -ne 'print if 16224..16482' file.txt > new_file.txt
I was about to post the head/tail trick, but actually I'd probably just fire up emacs. ;-)
open the new output file, ctl-y save
Let's me see what's happening.
Even we can do this to check at command line:
cat filename|sed 'n1,n2!d' > abc.txt
For Example:
cat foo.pl|sed '100,200!d' > abc.txt
# print section of file based on line numbers
sed -n '16224 ,16482p' # method 1
sed '16224,16482!d' # method 2
I wanted to do the same thing from a script using a variable and achieved it by putting quotes around the $variable to separate the variable name from the p:
sed -n "$first","$count"p imagelist.txt >"$imageblock"
I wanted to split a list into separate folders and found the initial question and answer a useful step. (split command not an option on the old os I have to port code to).
sed -n '16224,16482p' < dump.sql
There is another approach with awk:
awk 'NR==16224, NR==16482' file
If the file is huge, it can be good to exit after reading the last desired line. This way, it won't read the following lines unnecessarily:
awk 'NR==16224, NR==16482-1; NR==16482 {print; exit}' file
awk 'NR==16224, NR==16482; NR==16482 {exit}' file
You could use 'vi' and then the following command:
:16224,16482w!/tmp/some-file
Alternatively:
cat file | head -n 16482 | tail -n 258
EDIT:- Just to add explanation, you use head -n 16482 to display first 16482 lines then use tail -n 258 to get last 258 lines out of the first output.
Quick and dirty:
head -16428 < file.in | tail -259 > file.out
Probably not the best way to do it but it should work.
BTW: 259 = 16482-16224+1.
sed -n '16224,16482 p' orig-data-file > new-file
Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. -n suppresses echoing the input as output, which you clearly don't want; the numbers indicate the range of lines to make the following command operate on; the command p prints out the relevant lines.
I was looking for an answer to this but I had to end up writing my own code which worked. None of the answers above were satisfactory. Consider you have very large file and have certain line numbers that you want to print out but the numbers are not in order. You can do the following:
My relatively large file
for letter in {a..k} ; do echo $letter; done | cat -n > myfile.txt
1 a
2 b
3 c
4 d
5 e
6 f
7 g
8 h
9 i
10 j
11 k
Specific line numbers I want:
shuf -i 1-11 -n 4 > line_numbers_I_want.txt
10
11
4
9
To print these line numbers, do the following.
awk '{system("head myfile.txt -n " $0 " | tail -n 1")}' line_numbers_I_want.txt
What the above does is to head the n line then take the last line using tail
If you want your line numbers in order, sort ( is -n numeric sort) first then get the lines.
cat line_numbers_I_want.txt | sort -n | awk '{system("head myfile.txt -n " $0 " | tail -n 1")}'
4 d
9 i
10 j
11 k
cat dump.txt | head -16224 | tail -258
should do the trick. The downside of this approach is that you need to do the arithmetic to determine the argument for tail and to account for whether you want the 'between' to include the ending line or not.
I was about to post the head/tail trick, but actually I'd probably just fire up emacs. ;-)
open the new output file, ctl-y save
Let's me see what's happening.
Quick and dirty:
head -16428 < file.in | tail -259 > file.out
Probably not the best way to do it but it should work.
BTW: 259 = 16482-16224+1.
sed -n '16224,16482 p' orig-data-file > new-file
Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. -n suppresses echoing the input as output, which you clearly don't want; the numbers indicate the range of lines to make the following command operate on; the command p prints out the relevant lines.
Just benchmarking 3 solutions given above, that works to me:
Credits on the 3 solutions goes to:
I'm using a huge file I find in my server:
# wc fo2debug.1.log
10421186 19448208 38795491134 fo2debug.1.log
38 Gb in 10.4 million lines.
And yes, I have a logrotate problem. : ))
Getting 256 lines from the beginning of the file.
# time sed -n '1001,1256p;1256q' fo2debug.1.log | wc -l
256
real 0m0,003s
user 0m0,000s
sys 0m0,004s
# time head -1256 fo2debug.1.log | tail -n +1001 | wc -l
256
real 0m0,003s
user 0m0,006s
sys 0m0,000s
# time awk 'NR==1001, NR==1256; NR==1256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,002s
user 0m0,004s
sys 0m0,000s
Awk won. Technical tie in second place between sed and "head+tail".
Getting 256 lines at the end of the first third of the file.
# time sed -n '3473001,3473256p;3473256q' fo2debug.1.log | wc -l
256
real 0m0,265s
user 0m0,242s
sys 0m0,024s
# time head -3473256 fo2debug.1.log | tail -n +3473001 | wc -l
256
real 0m0,308s
user 0m0,313s
sys 0m0,145s
# time awk 'NR==3473001, NR==3473256; NR==3473256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,393s
user 0m0,326s
sys 0m0,068s
Sed won. Followed by "head+tail" and, finally, awk.
Getting 256 lines at the end of the second third of the file.
# time sed -n '6947001,6947256p;6947256q' fo2debug.1.log | wc -l
A256
real 0m0,525s
user 0m0,462s
sys 0m0,064s
# time head -6947256 fo2debug.1.log | tail -n +6947001 | wc -l
256
real 0m0,615s
user 0m0,488s
sys 0m0,423s
# time awk 'NR==6947001, NR==6947256; NR==6947256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,779s
user 0m0,650s
sys 0m0,130s
Same results.
Sed won. Followed by "head+tail" and, finally, awk.
Getting 256 lines near the end of the file.
# time sed -n '10420001,10420256p;10420256q' fo2debug.1.log | wc -l
256
real 1m50,017s
user 0m12,735s
sys 0m22,926s
# time head -10420256 fo2debug.1.log | tail -n +10420001 | wc -l
256
real 1m48,269s
user 0m42,404s
sys 0m51,015s
# time awk 'NR==10420001, NR==10420256; NR==10420256 {exit}' fo2debug.1.log | wc -l
256
real 1m49,106s
user 0m12,322s
sys 0m18,576s
And suddenly, a twist!
"Head+tail" won. Followed by awk and, finally, sed.
(some hours later...)
My analysis above ends up being an example of a basic flaw in doing an analysis.
The flaw is not knowing in depth the resources used for the analysis.
In this case, I used a log file to analyze the performance of a search for a certain number of lines within it.
Using 3 different techniques, searches were made at different points in the file, comparing the performance of the techniques at each point and checking whether the results varied depending on the point in the file where the search was made.
My mistake was to assume that there was a certain homogeneity of content in the log file.
The reality is that long lines appear more frequently at the end of the file.
Thus, the apparent conclusion that longer searches (closer to the end of the file) are better with a given technique, may be biased. In fact, this technique may be better when dealing with longer lines. What remains to be confirmed.
I wrote a small bash script that you can run from your command line, so long as you update your PATH to include its directory (or you can place it in a directory that is already contained in the PATH).
Usage: $ pinch filename start-line end-line
#!/bin/bash
# Display line number ranges of a file to the terminal.
# Usage: $ pinch filename start-line end-line
# By Evan J. Coon
FILENAME=$1
START=$2
END=$3
ERROR="[PINCH ERROR]"
# Check that the number of arguments is 3
if [ $# -lt 3 ]; then
echo "$ERROR Need three arguments: Filename Start-line End-line"
exit 1
fi
# Check that the file exists.
if [ ! -f "$FILENAME" ]; then
echo -e "$ERROR File does not exist. \n\t$FILENAME"
exit 1
fi
# Check that start-line is not greater than end-line
if [ "$START" -gt "$END" ]; then
echo -e "$ERROR Start line is greater than End line."
exit 1
fi
# Check that start-line is positive.
if [ "$START" -lt 0 ]; then
echo -e "$ERROR Start line is less than 0."
exit 1
fi
# Check that end-line is positive.
if [ "$END" -lt 0 ]; then
echo -e "$ERROR End line is less than 0."
exit 1
fi
NUMOFLINES=$(wc -l < "$FILENAME")
# Check that end-line is not greater than the number of lines in the file.
if [ "$END" -gt "$NUMOFLINES" ]; then
echo -e "$ERROR End line is greater than number of lines in file."
exit 1
fi
# The distance from the end of the file to end-line
ENDDIFF=$(( NUMOFLINES - END ))
# For larger files, this will run more quickly. If the distance from the
# end of the file to the end-line is less than the distance from the
# start of the file to the start-line, then start pinching from the
# bottom as opposed to the top.
if [ "$START" -lt "$ENDDIFF" ]; then
< "$FILENAME" head -n $END | tail -n +$START
else
< "$FILENAME" tail -n +$START | head -n $(( END-START+1 ))
fi
# Success
exit 0
Standing on the shoulders of boxxar, I like this:
sed -n '<first line>,$p;<last line>q' input
e.g.
sed -n '16224,$p;16482q' input
The $ means "last line", so the first command makes sed print all lines starting with line 16224 and the second command makes sed quit after printing line 16428. (Adding 1 for the q-range in boxxar's solution does not seem to be necessary.)
I like this variant because I don't need to specify the ending line number twice. And I measured that using $ does not have detrimental effects on performance.
I was about to post the head/tail trick, but actually I'd probably just fire up emacs. ;-)
open the new output file, ctl-y save
Let's me see what's happening.
You could use 'vi' and then the following command:
:16224,16482w!/tmp/some-file
Alternatively:
cat file | head -n 16482 | tail -n 258
EDIT:- Just to add explanation, you use head -n 16482 to display first 16482 lines then use tail -n 258 to get last 258 lines out of the first output.
# print section of file based on line numbers
sed -n '16224 ,16482p' # method 1
sed '16224,16482!d' # method 2
Just benchmarking 3 solutions given above, that works to me:
Credits on the 3 solutions goes to:
I'm using a huge file I find in my server:
# wc fo2debug.1.log
10421186 19448208 38795491134 fo2debug.1.log
38 Gb in 10.4 million lines.
And yes, I have a logrotate problem. : ))
Getting 256 lines from the beginning of the file.
# time sed -n '1001,1256p;1256q' fo2debug.1.log | wc -l
256
real 0m0,003s
user 0m0,000s
sys 0m0,004s
# time head -1256 fo2debug.1.log | tail -n +1001 | wc -l
256
real 0m0,003s
user 0m0,006s
sys 0m0,000s
# time awk 'NR==1001, NR==1256; NR==1256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,002s
user 0m0,004s
sys 0m0,000s
Awk won. Technical tie in second place between sed and "head+tail".
Getting 256 lines at the end of the first third of the file.
# time sed -n '3473001,3473256p;3473256q' fo2debug.1.log | wc -l
256
real 0m0,265s
user 0m0,242s
sys 0m0,024s
# time head -3473256 fo2debug.1.log | tail -n +3473001 | wc -l
256
real 0m0,308s
user 0m0,313s
sys 0m0,145s
# time awk 'NR==3473001, NR==3473256; NR==3473256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,393s
user 0m0,326s
sys 0m0,068s
Sed won. Followed by "head+tail" and, finally, awk.
Getting 256 lines at the end of the second third of the file.
# time sed -n '6947001,6947256p;6947256q' fo2debug.1.log | wc -l
A256
real 0m0,525s
user 0m0,462s
sys 0m0,064s
# time head -6947256 fo2debug.1.log | tail -n +6947001 | wc -l
256
real 0m0,615s
user 0m0,488s
sys 0m0,423s
# time awk 'NR==6947001, NR==6947256; NR==6947256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,779s
user 0m0,650s
sys 0m0,130s
Same results.
Sed won. Followed by "head+tail" and, finally, awk.
Getting 256 lines near the end of the file.
# time sed -n '10420001,10420256p;10420256q' fo2debug.1.log | wc -l
256
real 1m50,017s
user 0m12,735s
sys 0m22,926s
# time head -10420256 fo2debug.1.log | tail -n +10420001 | wc -l
256
real 1m48,269s
user 0m42,404s
sys 0m51,015s
# time awk 'NR==10420001, NR==10420256; NR==10420256 {exit}' fo2debug.1.log | wc -l
256
real 1m49,106s
user 0m12,322s
sys 0m18,576s
And suddenly, a twist!
"Head+tail" won. Followed by awk and, finally, sed.
(some hours later...)
My analysis above ends up being an example of a basic flaw in doing an analysis.
The flaw is not knowing in depth the resources used for the analysis.
In this case, I used a log file to analyze the performance of a search for a certain number of lines within it.
Using 3 different techniques, searches were made at different points in the file, comparing the performance of the techniques at each point and checking whether the results varied depending on the point in the file where the search was made.
My mistake was to assume that there was a certain homogeneity of content in the log file.
The reality is that long lines appear more frequently at the end of the file.
Thus, the apparent conclusion that longer searches (closer to the end of the file) are better with a given technique, may be biased. In fact, this technique may be better when dealing with longer lines. What remains to be confirmed.
# print section of file based on line numbers
sed -n '16224 ,16482p' # method 1
sed '16224,16482!d' # method 2
Using ruby:
ruby -ne 'puts "#{$.}: #{$_}" if $. >= 32613500 && $. <= 32614500' < GND.rdf > GND.extract.rdf
# print section of file based on line numbers
sed -n '16224 ,16482p' # method 1
sed '16224,16482!d' # method 2
The -n in the accept answers work. Here's another way in case you're inclined.
cat $filename | sed "${linenum}p;d";
This does the following:
sed -n '16224,16482 p' orig-data-file > new-file
Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. -n suppresses echoing the input as output, which you clearly don't want; the numbers indicate the range of lines to make the following command operate on; the command p prints out the relevant lines.
sed -n '16224,16482p' < dump.sql
I was about to post the head/tail trick, but actually I'd probably just fire up emacs. ;-)
open the new output file, ctl-y save
Let's me see what's happening.
Quick and dirty:
head -16428 < file.in | tail -259 > file.out
Probably not the best way to do it but it should work.
BTW: 259 = 16482-16224+1.
cat dump.txt | head -16224 | tail -258
should do the trick. The downside of this approach is that you need to do the arithmetic to determine the argument for tail and to account for whether you want the 'between' to include the ending line or not.
sed -n '16224,16482 p' orig-data-file > new-file
Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. -n suppresses echoing the input as output, which you clearly don't want; the numbers indicate the range of lines to make the following command operate on; the command p prints out the relevant lines.
I wanted to do the same thing from a script using a variable and achieved it by putting quotes around the $variable to separate the variable name from the p:
sed -n "$first","$count"p imagelist.txt >"$imageblock"
I wanted to split a list into separate folders and found the initial question and answer a useful step. (split command not an option on the old os I have to port code to).
sed -n '16224,16482p' < dump.sql
sed -n '16224,16482 p' orig-data-file > new-file
Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. -n suppresses echoing the input as output, which you clearly don't want; the numbers indicate the range of lines to make the following command operate on; the command p prints out the relevant lines.
I wanted to do the same thing from a script using a variable and achieved it by putting quotes around the $variable to separate the variable name from the p:
sed -n "$first","$count"p imagelist.txt >"$imageblock"
I wanted to split a list into separate folders and found the initial question and answer a useful step. (split command not an option on the old os I have to port code to).
Since we are talking about extracting lines of text from a text file, I will give an special case where you want to extract all lines that match a certain pattern.
myfile content:
=====================
line1 not needed
line2 also discarded
[Data]
first data line
second data line
=====================
sed -n '/Data/,$p' myfile
Will print the [Data] line and the remaining. If you want the text from line1 to the pattern, you type: sed -n '1,/Data/p' myfile. Furthermore, if you know two pattern (better be unique in your text), both the beginning and end line of the range can be specified with matches.
sed -n '/BEGIN_MARK/,/END_MARK/p' myfile
I wrote a Haskell program called splitter that does exactly this: have a read through my release blog post.
You can use the program as follows:
$ cat somefile | splitter 16224-16482
And that is all that there is to it. You will need Haskell to install it. Just:
$ cabal install splitter
And you are done. I hope that you find this program useful.
Standing on the shoulders of boxxar, I like this:
sed -n '<first line>,$p;<last line>q' input
e.g.
sed -n '16224,$p;16482q' input
The $ means "last line", so the first command makes sed print all lines starting with line 16224 and the second command makes sed quit after printing line 16428. (Adding 1 for the q-range in boxxar's solution does not seem to be necessary.)
I like this variant because I don't need to specify the ending line number twice. And I measured that using $ does not have detrimental effects on performance.
Even we can do this to check at command line:
cat filename|sed 'n1,n2!d' > abc.txt
For Example:
cat foo.pl|sed '100,200!d' > abc.txt
I wrote a Haskell program called splitter that does exactly this: have a read through my release blog post.
You can use the program as follows:
$ cat somefile | splitter 16224-16482
And that is all that there is to it. You will need Haskell to install it. Just:
$ cabal install splitter
And you are done. I hope that you find this program useful.
There is another approach with awk:
awk 'NR==16224, NR==16482' file
If the file is huge, it can be good to exit after reading the last desired line. This way, it won't read the following lines unnecessarily:
awk 'NR==16224, NR==16482-1; NR==16482 {print; exit}' file
awk 'NR==16224, NR==16482; NR==16482 {exit}' file
sed -n '16224,16482p' < dump.sql
Using ruby:
ruby -ne 'puts "#{$.}: #{$_}" if $. >= 32613500 && $. <= 32614500' < GND.rdf > GND.extract.rdf
I was looking for an answer to this but I had to end up writing my own code which worked. None of the answers above were satisfactory. Consider you have very large file and have certain line numbers that you want to print out but the numbers are not in order. You can do the following:
My relatively large file
for letter in {a..k} ; do echo $letter; done | cat -n > myfile.txt
1 a
2 b
3 c
4 d
5 e
6 f
7 g
8 h
9 i
10 j
11 k
Specific line numbers I want:
shuf -i 1-11 -n 4 > line_numbers_I_want.txt
10
11
4
9
To print these line numbers, do the following.
awk '{system("head myfile.txt -n " $0 " | tail -n 1")}' line_numbers_I_want.txt
What the above does is to head the n line then take the last line using tail
If you want your line numbers in order, sort ( is -n numeric sort) first then get the lines.
cat line_numbers_I_want.txt | sort -n | awk '{system("head myfile.txt -n " $0 " | tail -n 1")}'
4 d
9 i
10 j
11 k
sed -n '16224,16482p' < dump.sql
There is another approach with awk:
awk 'NR==16224, NR==16482' file
If the file is huge, it can be good to exit after reading the last desired line. This way, it won't read the following lines unnecessarily:
awk 'NR==16224, NR==16482-1; NR==16482 {print; exit}' file
awk 'NR==16224, NR==16482; NR==16482 {exit}' file
Using ed:
ed -s infile <<<'16224,16482p'
-s suppresses diagnostic output; the actual commands are in a here-string. Specifically, 16224,16482p runs the p (print) command on the desired line address range.
The -n in the accept answers work. Here's another way in case you're inclined.
cat $filename | sed "${linenum}p;d";
This does the following:
I wrote a small bash script that you can run from your command line, so long as you update your PATH to include its directory (or you can place it in a directory that is already contained in the PATH).
Usage: $ pinch filename start-line end-line
#!/bin/bash
# Display line number ranges of a file to the terminal.
# Usage: $ pinch filename start-line end-line
# By Evan J. Coon
FILENAME=$1
START=$2
END=$3
ERROR="[PINCH ERROR]"
# Check that the number of arguments is 3
if [ $# -lt 3 ]; then
echo "$ERROR Need three arguments: Filename Start-line End-line"
exit 1
fi
# Check that the file exists.
if [ ! -f "$FILENAME" ]; then
echo -e "$ERROR File does not exist. \n\t$FILENAME"
exit 1
fi
# Check that start-line is not greater than end-line
if [ "$START" -gt "$END" ]; then
echo -e "$ERROR Start line is greater than End line."
exit 1
fi
# Check that start-line is positive.
if [ "$START" -lt 0 ]; then
echo -e "$ERROR Start line is less than 0."
exit 1
fi
# Check that end-line is positive.
if [ "$END" -lt 0 ]; then
echo -e "$ERROR End line is less than 0."
exit 1
fi
NUMOFLINES=$(wc -l < "$FILENAME")
# Check that end-line is not greater than the number of lines in the file.
if [ "$END" -gt "$NUMOFLINES" ]; then
echo -e "$ERROR End line is greater than number of lines in file."
exit 1
fi
# The distance from the end of the file to end-line
ENDDIFF=$(( NUMOFLINES - END ))
# For larger files, this will run more quickly. If the distance from the
# end of the file to the end-line is less than the distance from the
# start of the file to the start-line, then start pinching from the
# bottom as opposed to the top.
if [ "$START" -lt "$ENDDIFF" ]; then
< "$FILENAME" head -n $END | tail -n +$START
else
< "$FILENAME" tail -n +$START | head -n $(( END-START+1 ))
fi
# Success
exit 0
The -n in the accept answers work. Here's another way in case you're inclined.
cat $filename | sed "${linenum}p;d";
This does the following:
Since we are talking about extracting lines of text from a text file, I will give an special case where you want to extract all lines that match a certain pattern.
myfile content:
=====================
line1 not needed
line2 also discarded
[Data]
first data line
second data line
=====================
sed -n '/Data/,$p' myfile
Will print the [Data] line and the remaining. If you want the text from line1 to the pattern, you type: sed -n '1,/Data/p' myfile. Furthermore, if you know two pattern (better be unique in your text), both the beginning and end line of the range can be specified with matches.
sed -n '/BEGIN_MARK/,/END_MARK/p' myfile
I would use:
awk 'FNR >= 16224 && FNR <= 16482' my_file > extracted.txt
FNR contains the record (line) number of the line being read from the file.
cat dump.txt | head -16224 | tail -258
should do the trick. The downside of this approach is that you need to do the arithmetic to determine the argument for tail and to account for whether you want the 'between' to include the ending line or not.
perl -ne 'print if 16224..16482' file.txt > new_file.txt
# print section of file based on line numbers
sed -n '16224 ,16482p' # method 1
sed '16224,16482!d' # method 2
Even we can do this to check at command line:
cat filename|sed 'n1,n2!d' > abc.txt
For Example:
cat foo.pl|sed '100,200!d' > abc.txt
sed -n '16224,16482p' < dump.sql
Using ed:
ed -s infile <<<'16224,16482p'
-s suppresses diagnostic output; the actual commands are in a here-string. Specifically, 16224,16482p runs the p (print) command on the desired line address range.
I was about to post the head/tail trick, but actually I'd probably just fire up emacs. ;-)
open the new output file, ctl-y save
Let's me see what's happening.
Quick and dirty:
head -16428 < file.in | tail -259 > file.out
Probably not the best way to do it but it should work.
BTW: 259 = 16482-16224+1.
I wrote a small bash script that you can run from your command line, so long as you update your PATH to include its directory (or you can place it in a directory that is already contained in the PATH).
Usage: $ pinch filename start-line end-line
#!/bin/bash
# Display line number ranges of a file to the terminal.
# Usage: $ pinch filename start-line end-line
# By Evan J. Coon
FILENAME=$1
START=$2
END=$3
ERROR="[PINCH ERROR]"
# Check that the number of arguments is 3
if [ $# -lt 3 ]; then
echo "$ERROR Need three arguments: Filename Start-line End-line"
exit 1
fi
# Check that the file exists.
if [ ! -f "$FILENAME" ]; then
echo -e "$ERROR File does not exist. \n\t$FILENAME"
exit 1
fi
# Check that start-line is not greater than end-line
if [ "$START" -gt "$END" ]; then
echo -e "$ERROR Start line is greater than End line."
exit 1
fi
# Check that start-line is positive.
if [ "$START" -lt 0 ]; then
echo -e "$ERROR Start line is less than 0."
exit 1
fi
# Check that end-line is positive.
if [ "$END" -lt 0 ]; then
echo -e "$ERROR End line is less than 0."
exit 1
fi
NUMOFLINES=$(wc -l < "$FILENAME")
# Check that end-line is not greater than the number of lines in the file.
if [ "$END" -gt "$NUMOFLINES" ]; then
echo -e "$ERROR End line is greater than number of lines in file."
exit 1
fi
# The distance from the end of the file to end-line
ENDDIFF=$(( NUMOFLINES - END ))
# For larger files, this will run more quickly. If the distance from the
# end of the file to the end-line is less than the distance from the
# start of the file to the start-line, then start pinching from the
# bottom as opposed to the top.
if [ "$START" -lt "$ENDDIFF" ]; then
< "$FILENAME" head -n $END | tail -n +$START
else
< "$FILENAME" tail -n +$START | head -n $(( END-START+1 ))
fi
# Success
exit 0
There is another approach with awk:
awk 'NR==16224, NR==16482' file
If the file is huge, it can be good to exit after reading the last desired line. This way, it won't read the following lines unnecessarily:
awk 'NR==16224, NR==16482-1; NR==16482 {print; exit}' file
awk 'NR==16224, NR==16482; NR==16482 {exit}' file
cat dump.txt | head -16224 | tail -258
should do the trick. The downside of this approach is that you need to do the arithmetic to determine the argument for tail and to account for whether you want the 'between' to include the ending line or not.
Quite simple using head/tail:
head -16482 in.sql | tail -258 > out.sql
using sed:
sed -n '16224,16482p' in.sql > out.sql
using awk:
awk 'NR>=16224&&NR<=16482' in.sql > out.sql
perl -ne 'print if 16224..16482' file.txt > new_file.txt
I would use:
awk 'FNR >= 16224 && FNR <= 16482' my_file > extracted.txt
FNR contains the record (line) number of the line being read from the file.
I was about to post the head/tail trick, but actually I'd probably just fire up emacs. ;-)
open the new output file, ctl-y save
Let's me see what's happening.
I wrote a Haskell program called splitter that does exactly this: have a read through my release blog post.
You can use the program as follows:
$ cat somefile | splitter 16224-16482
And that is all that there is to it. You will need Haskell to install it. Just:
$ cabal install splitter
And you are done. I hope that you find this program useful.
cat dump.txt | head -16224 | tail -258
should do the trick. The downside of this approach is that you need to do the arithmetic to determine the argument for tail and to account for whether you want the 'between' to include the ending line or not.
You could use 'vi' and then the following command:
:16224,16482w!/tmp/some-file
Alternatively:
cat file | head -n 16482 | tail -n 258
EDIT:- Just to add explanation, you use head -n 16482 to display first 16482 lines then use tail -n 258 to get last 258 lines out of the first output.
Standing on the shoulders of boxxar, I like this:
sed -n '<first line>,$p;<last line>q' input
e.g.
sed -n '16224,$p;16482q' input
The $ means "last line", so the first command makes sed print all lines starting with line 16224 and the second command makes sed quit after printing line 16428. (Adding 1 for the q-range in boxxar's solution does not seem to be necessary.)
I like this variant because I don't need to specify the ending line number twice. And I measured that using $ does not have detrimental effects on performance.
Quite simple using head/tail:
head -16482 in.sql | tail -258 > out.sql
using sed:
sed -n '16224,16482p' in.sql > out.sql
using awk:
awk 'NR>=16224&&NR<=16482' in.sql > out.sql
This might work for you (GNU sed):
sed -ne '16224,16482w newfile' -e '16482q' file
or taking advantage of bash:
sed -n $'16224,16482w newfile\n16482q' file
Since we are talking about extracting lines of text from a text file, I will give an special case where you want to extract all lines that match a certain pattern.
myfile content:
=====================
line1 not needed
line2 also discarded
[Data]
first data line
second data line
=====================
sed -n '/Data/,$p' myfile
Will print the [Data] line and the remaining. If you want the text from line1 to the pattern, you type: sed -n '1,/Data/p' myfile. Furthermore, if you know two pattern (better be unique in your text), both the beginning and end line of the range can be specified with matches.
sed -n '/BEGIN_MARK/,/END_MARK/p' myfile
Using ed:
ed -s infile <<<'16224,16482p'
-s suppresses diagnostic output; the actual commands are in a here-string. Specifically, 16224,16482p runs the p (print) command on the desired line address range.
Quick and dirty:
head -16428 < file.in | tail -259 > file.out
Probably not the best way to do it but it should work.
BTW: 259 = 16482-16224+1.
Quite simple using head/tail:
head -16482 in.sql | tail -258 > out.sql
using sed:
sed -n '16224,16482p' in.sql > out.sql
using awk:
awk 'NR>=16224&&NR<=16482' in.sql > out.sql
perl -ne 'print if 16224..16482' file.txt > new_file.txt
I wanted to do the same thing from a script using a variable and achieved it by putting quotes around the $variable to separate the variable name from the p:
sed -n "$first","$count"p imagelist.txt >"$imageblock"
I wanted to split a list into separate folders and found the initial question and answer a useful step. (split command not an option on the old os I have to port code to).
Since we are talking about extracting lines of text from a text file, I will give an special case where you want to extract all lines that match a certain pattern.
myfile content:
=====================
line1 not needed
line2 also discarded
[Data]
first data line
second data line
=====================
sed -n '/Data/,$p' myfile
Will print the [Data] line and the remaining. If you want the text from line1 to the pattern, you type: sed -n '1,/Data/p' myfile. Furthermore, if you know two pattern (better be unique in your text), both the beginning and end line of the range can be specified with matches.
sed -n '/BEGIN_MARK/,/END_MARK/p' myfile
cat dump.txt | head -16224 | tail -258
should do the trick. The downside of this approach is that you need to do the arithmetic to determine the argument for tail and to account for whether you want the 'between' to include the ending line or not.
Quick and dirty:
head -16428 < file.in | tail -259 > file.out
Probably not the best way to do it but it should work.
BTW: 259 = 16482-16224+1.
perl -ne 'print if 16224..16482' file.txt > new_file.txt
Quick and dirty:
head -16428 < file.in | tail -259 > file.out
Probably not the best way to do it but it should work.
BTW: 259 = 16482-16224+1.
sed -n '16224,16482p' < dump.sql
# print section of file based on line numbers
sed -n '16224 ,16482p' # method 1
sed '16224,16482!d' # method 2
perl -ne 'print if 16224..16482' file.txt > new_file.txt
sed -n '16224,16482 p' orig-data-file > new-file
Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. -n suppresses echoing the input as output, which you clearly don't want; the numbers indicate the range of lines to make the following command operate on; the command p prints out the relevant lines.
This might work for you (GNU sed):
sed -ne '16224,16482w newfile' -e '16482q' file
or taking advantage of bash:
sed -n $'16224,16482w newfile\n16482q' file
Using ruby:
ruby -ne 'puts "#{$.}: #{$_}" if $. >= 32613500 && $. <= 32614500' < GND.rdf > GND.extract.rdf
Quite simple using head/tail:
head -16482 in.sql | tail -258 > out.sql
using sed:
sed -n '16224,16482p' in.sql > out.sql
using awk:
awk 'NR>=16224&&NR<=16482' in.sql > out.sql
I wrote a small bash script that you can run from your command line, so long as you update your PATH to include its directory (or you can place it in a directory that is already contained in the PATH).
Usage: $ pinch filename start-line end-line
#!/bin/bash
# Display line number ranges of a file to the terminal.
# Usage: $ pinch filename start-line end-line
# By Evan J. Coon
FILENAME=$1
START=$2
END=$3
ERROR="[PINCH ERROR]"
# Check that the number of arguments is 3
if [ $# -lt 3 ]; then
echo "$ERROR Need three arguments: Filename Start-line End-line"
exit 1
fi
# Check that the file exists.
if [ ! -f "$FILENAME" ]; then
echo -e "$ERROR File does not exist. \n\t$FILENAME"
exit 1
fi
# Check that start-line is not greater than end-line
if [ "$START" -gt "$END" ]; then
echo -e "$ERROR Start line is greater than End line."
exit 1
fi
# Check that start-line is positive.
if [ "$START" -lt 0 ]; then
echo -e "$ERROR Start line is less than 0."
exit 1
fi
# Check that end-line is positive.
if [ "$END" -lt 0 ]; then
echo -e "$ERROR End line is less than 0."
exit 1
fi
NUMOFLINES=$(wc -l < "$FILENAME")
# Check that end-line is not greater than the number of lines in the file.
if [ "$END" -gt "$NUMOFLINES" ]; then
echo -e "$ERROR End line is greater than number of lines in file."
exit 1
fi
# The distance from the end of the file to end-line
ENDDIFF=$(( NUMOFLINES - END ))
# For larger files, this will run more quickly. If the distance from the
# end of the file to the end-line is less than the distance from the
# start of the file to the start-line, then start pinching from the
# bottom as opposed to the top.
if [ "$START" -lt "$ENDDIFF" ]; then
< "$FILENAME" head -n $END | tail -n +$START
else
< "$FILENAME" tail -n +$START | head -n $(( END-START+1 ))
fi
# Success
exit 0
You could use 'vi' and then the following command:
:16224,16482w!/tmp/some-file
Alternatively:
cat file | head -n 16482 | tail -n 258
EDIT:- Just to add explanation, you use head -n 16482 to display first 16482 lines then use tail -n 258 to get last 258 lines out of the first output.
This might work for you (GNU sed):
sed -ne '16224,16482w newfile' -e '16482q' file
or taking advantage of bash:
sed -n $'16224,16482w newfile\n16482q' file
Quite simple using head/tail:
head -16482 in.sql | tail -258 > out.sql
using sed:
sed -n '16224,16482p' in.sql > out.sql
using awk:
awk 'NR>=16224&&NR<=16482' in.sql > out.sql
This might work for you (GNU sed):
sed -ne '16224,16482w newfile' -e '16482q' file
or taking advantage of bash:
sed -n $'16224,16482w newfile\n16482q' file
I would use:
awk 'FNR >= 16224 && FNR <= 16482' my_file > extracted.txt
FNR contains the record (line) number of the line being read from the file.
Using ruby:
ruby -ne 'puts "#{$.}: #{$_}" if $. >= 32613500 && $. <= 32614500' < GND.rdf > GND.extract.rdf
I would use:
awk 'FNR >= 16224 && FNR <= 16482' my_file > extracted.txt
FNR contains the record (line) number of the line being read from the file.
Just benchmarking 3 solutions given above, that works to me:
Credits on the 3 solutions goes to:
I'm using a huge file I find in my server:
# wc fo2debug.1.log
10421186 19448208 38795491134 fo2debug.1.log
38 Gb in 10.4 million lines.
And yes, I have a logrotate problem. : ))
Getting 256 lines from the beginning of the file.
# time sed -n '1001,1256p;1256q' fo2debug.1.log | wc -l
256
real 0m0,003s
user 0m0,000s
sys 0m0,004s
# time head -1256 fo2debug.1.log | tail -n +1001 | wc -l
256
real 0m0,003s
user 0m0,006s
sys 0m0,000s
# time awk 'NR==1001, NR==1256; NR==1256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,002s
user 0m0,004s
sys 0m0,000s
Awk won. Technical tie in second place between sed and "head+tail".
Getting 256 lines at the end of the first third of the file.
# time sed -n '3473001,3473256p;3473256q' fo2debug.1.log | wc -l
256
real 0m0,265s
user 0m0,242s
sys 0m0,024s
# time head -3473256 fo2debug.1.log | tail -n +3473001 | wc -l
256
real 0m0,308s
user 0m0,313s
sys 0m0,145s
# time awk 'NR==3473001, NR==3473256; NR==3473256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,393s
user 0m0,326s
sys 0m0,068s
Sed won. Followed by "head+tail" and, finally, awk.
Getting 256 lines at the end of the second third of the file.
# time sed -n '6947001,6947256p;6947256q' fo2debug.1.log | wc -l
A256
real 0m0,525s
user 0m0,462s
sys 0m0,064s
# time head -6947256 fo2debug.1.log | tail -n +6947001 | wc -l
256
real 0m0,615s
user 0m0,488s
sys 0m0,423s
# time awk 'NR==6947001, NR==6947256; NR==6947256 {exit}' fo2debug.1.log | wc -l
256
real 0m0,779s
user 0m0,650s
sys 0m0,130s
Same results.
Sed won. Followed by "head+tail" and, finally, awk.
Getting 256 lines near the end of the file.
# time sed -n '10420001,10420256p;10420256q' fo2debug.1.log | wc -l
256
real 1m50,017s
user 0m12,735s
sys 0m22,926s
# time head -10420256 fo2debug.1.log | tail -n +10420001 | wc -l
256
real 1m48,269s
user 0m42,404s
sys 0m51,015s
# time awk 'NR==10420001, NR==10420256; NR==10420256 {exit}' fo2debug.1.log | wc -l
256
real 1m49,106s
user 0m12,322s
sys 0m18,576s
And suddenly, a twist!
"Head+tail" won. Followed by awk and, finally, sed.
(some hours later...)
My analysis above ends up being an example of a basic flaw in doing an analysis.
The flaw is not knowing in depth the resources used for the analysis.
In this case, I used a log file to analyze the performance of a search for a certain number of lines within it.
Using 3 different techniques, searches were made at different points in the file, comparing the performance of the techniques at each point and checking whether the results varied depending on the point in the file where the search was made.
My mistake was to assume that there was a certain homogeneity of content in the log file.
The reality is that long lines appear more frequently at the end of the file.
Thus, the apparent conclusion that longer searches (closer to the end of the file) are better with a given technique, may be biased. In fact, this technique may be better when dealing with longer lines. What remains to be confirmed.
Standing on the shoulders of boxxar, I like this:
sed -n '<first line>,$p;<last line>q' input
e.g.
sed -n '16224,$p;16482q' input
The $ means "last line", so the first command makes sed print all lines starting with line 16224 and the second command makes sed quit after printing line 16428. (Adding 1 for the q-range in boxxar's solution does not seem to be necessary.)
I like this variant because I don't need to specify the ending line number twice. And I measured that using $ does not have detrimental effects on performance.
You could use 'vi' and then the following command:
:16224,16482w!/tmp/some-file
Alternatively:
cat file | head -n 16482 | tail -n 258
EDIT:- Just to add explanation, you use head -n 16482 to display first 16482 lines then use tail -n 258 to get last 258 lines out of the first output.
I wrote a Haskell program called splitter that does exactly this: have a read through my release blog post.
You can use the program as follows:
$ cat somefile | splitter 16224-16482
And that is all that there is to it. You will need Haskell to install it. Just:
$ cabal install splitter
And you are done. I hope that you find this program useful.
Source: Stackoverflow.com