[bash] How to merge every two lines into one from the command line?

I have a text file with the following format. The first line is the "KEY" and the second line is the "VALUE".

KEY 4048:1736 string
3
KEY 0:1772 string
1
KEY 4192:1349 string
1
KEY 7329:2407 string
2
KEY 0:1774 string
1

I need the value in the same line as of the key. So the output should look like this...

KEY 4048:1736 string 3
KEY 0:1772 string 1
KEY 4192:1349 string 1
KEY 7329:2407 string 2
KEY 0:1774 string 1

It will be better if I could use some delimiter like $ or ,:

KEY 4048:1736 string , 3

How do I merge two lines into one?

If Perl is an option, you can try:

perl -0pe 's/(.*)\n(.*)\n/$1 $2\n/g' file.txt

You can also use the following vi command:

:%g/.*/j

Another solutions using vim (just for reference).

Solution 1:

Open file in vim vim filename, then execute command :% normal Jj

This command is very easy to understand:

• % : for all the lines,
• normal : execute normal command
• Jj : execute Join command, then jump to below line

After that, save the file and exit with :wq

Solution 2:

Execute the command in shell, vim -c ":% normal Jj" filename, then save the file and exit with :wq.

A slight variation on glenn jackman's answer using paste: if the value for the -d delimiter option contains more than one character, paste cycles through the characters one by one, and combined with the -s options keeps doing that while processing the same input file.

This means that we can use whatever we want to have as the separator plus the escape sequence \n to merge two lines at a time.

Using a comma:

$ paste -s -d ',\n' infile
KEY 4048:1736 string,3
KEY 0:1772 string,1
KEY 4192:1349 string,1
KEY 7329:2407 string,2
KEY 0:1774 string,1

and the dollar sign:

$ paste -s -d '$\n' infile
KEY 4048:1736 string$3
KEY 0:1772 string$1
KEY 4192:1349 string$1
KEY 7329:2407 string$2
KEY 0:1774 string$1

What this cannot do is use a separator consisting of multiple characters.

As a bonus, if the paste is POSIX compliant, this won't modify the newline of the last line in the file, so for an input file with an odd number of lines like

KEY 4048:1736 string
3
KEY 0:1772 string

paste won't tack on the separation character on the last line:

$ paste -s -d ',\n' infile
KEY 4048:1736 string,3
KEY 0:1772 string

nawk '$0 ~ /string$/ {printf "%s ",$0; getline; printf "%s\n", $0}' filename

This reads as

$0 ~ /string$/  ## matches any lines that end with the word string
printf          ## so print the first line without newline
getline         ## get the next line
printf "%s\n"   ## print the whole line and carriage return

You can use awk like this to combine ever 2 pair of lines:

awk '{ if (NR%2 != 0) line=$0; else {printf("%s %s\n", line, $0); line="";} } \
     END {if (length(line)) print line;}' flle

Simplest way is here:

1. Remove even lines and write it in some temp file 1.
2. Remove odd lines and write it in some temp file 2.
3. Combine two files in one by using paste command with -d (means delete space)

sed '0~2d' file > 1 && sed '1~2d' file > 2 && paste -d " " 1 2

A more-general solution (allows for more than one follow-up line to be joined) as a shell script. This adds a line between each, because I needed visibility, but that is easily remedied. This example is where the "key" line ended in : and no other lines did.

#!/bin/bash
#
# join "The rest of the story" when the first line of each   story
# matches $PATTERN
# Nice for looking for specific changes in bart output
#

PATTERN='*:';
LINEOUT=""
while read line; do
    case $line in
        $PATTERN)
                echo ""
                echo $LINEOUT
                LINEOUT="$line"
                        ;;
        "")
                LINEOUT=""
                echo ""
                ;;

        *)      LINEOUT="$LINEOUT $line"
                ;;
    esac        
done

Here is another way with awk:

awk 'ORS=NR%2?FS:RS' file

$ cat file
KEY 4048:1736 string
3
KEY 0:1772 string
1
KEY 4192:1349 string
1
KEY 7329:2407 string
2
KEY 0:1774 string
1

$ awk 'ORS=NR%2?FS:RS' file
KEY 4048:1736 string 3
KEY 0:1772 string 1
KEY 4192:1349 string 1
KEY 7329:2407 string 2
KEY 0:1774 string 1

As indicated by Ed Morton in the comments, it is better to add braces for safety and parens for portability.

awk '{ ORS = (NR%2 ? FS : RS) } 1' file

ORS stands for Output Record Separator. What we are doing here is testing a condition using the NR which stores the line number. If the modulo of NR is a true value (>0) then we set the Output Field Separator to the value of FS (Field Separator) which by default is space, else we assign the value of RS (Record Separator) which is newline.

If you wish to add , as the separator then use the following:

awk '{ ORS = (NR%2 ? "," : RS) } 1' file

Alternative to sed, awk, grep:

xargs -n2 -d'\n'

This is best when you want to join N lines and you only need space delimited output.

My original answer was xargs -n2 which separates on words rather than lines. -d can be used to split the input by any single character.

"ex" is a scriptable line editor that is in the same family as sed, awk, grep, etc. I think it might be what you are looking for. Many modern vi clone/successors also have a vi mode.

 ex -c "%g/KEY/j" -c "wq" data.txt

This says for each line, if it matches "KEY" perform a j oin of the following line. After that command completes (against all lines), issue a w rite and q uit.

Although it seems the previous solutions would work, if a single anomaly occurs in the document the output would go to pieces. Below is a bit safer.

sed -n '/KEY/{
N
s/\n/ /p
}' somefile.txt

There are more ways to kill a dog than hanging. ^[1]

awk '{key=$0; getline; print key ", " $0;}'

Put whatever delimiter you like inside the quotes.

References:

1. Originally "Plenty of ways to skin the cat", reverted to an older, potentially originating expression that also has nothing to do with pets.

Try the following line:

while read line1; do read line2; echo "$line1 $line2"; done <old.txt>new_file

Put delimiter in-between

"$line1 $line2";

e.g. if the delimiter is |, then:

"$line1|$line2";

You can use xargs like this:

xargs -a file

perl -0pE 's{^KEY.*?\K\s+(\d+)$}{ $1}msg;' data.txt > data_merged-lines.txt

-0 gobbles the whole file instead of reading it line-by-line;
pE wraps code with loop and prints the output, see details in http://perldoc.perl.org/perlrun.html;
^KEY match "KEY" in the beginning of line, followed by non-greedy match of anything (.*?) before sequence of

1. one or more spaces \s+ of any kind including line breaks;
2. one or more digit (\d+) which we capture and later re-insert as $1;

followed by the end of line $.

\K conveniently excludes everything on its left hand side from substitution so { $1} replaces only 1-2 sequence, see http://perldoc.perl.org/perlre.html.

Here is my solution in bash:

while read line1; do read line2; echo "$line1, $line2"; done < data.txt

Another approach using vim would be:

:g/KEY/join

This applies a join (to the line below it) to all lines that have the word KEY in it. Result:

KEY 4048:1736 string 3
KEY 0:1772 string 1
KEY 4192:1349 string 1
KEY 7329:2407 string 2
KEY 0:1774 string 1

In the case where I needed to combine two lines (for easier processing), but allow the data past the specific, I found this to be useful

data.txt

string1=x
string2=y
string3
string4

cat data.txt | nawk '$0 ~ /string1=/ { printf "%s ", $0; getline; printf "%s\n", $0; getline } { print }' > converted_data.txt

output then looks like:

converted_data.txt

string1=x string2=y
string3
string4

paste is good for this job:

paste -d " "  - - < filename