[sed] How can I replace a newline (\n) using sed?

How can I replace a newline ("\n") with a space ("") using the sed command?

I unsuccessfully tried:

sed 's#\n# #g' file
sed 's#^$# #g' file

How do I fix it?

Here is sed without buffers (good for real time output).
Example: replacing \n with <br/> break in HTML

echo -e "1\n2\n3" | sed 's/.*$/&<br\/>/'

You can use xargs:

seq 10 | xargs

or

seq 10 | xargs echo -n

In order to replace all newlines with spaces using awk, without reading the whole file into memory:

awk '{printf "%s ", $0}' inputfile

If you want a final newline:

awk '{printf "%s ", $0} END {printf "\n"}' inputfile

You can use a character other than space:

awk '{printf "%s|", $0} END {printf "\n"}' inputfile

Who needs sed? Here is the bash way:

cat test.txt |  while read line; do echo -n "$line "; done

Using Awk:

awk "BEGIN { o=\"\" }  { o=o \" \" \$0 }  END { print o; }"

You can also use the Standard Text Editor:

printf '%s\n%s\n%s\n' '%s/$/ /' '%j' 'w' | ed -s file

Note: this saves the result back to file.

As with sed, this solution suffers from having to load the whole file into memory first.

sed is intended to be used on line-based input. Although it can do what you need.

A better option here is to use the tr command as follows:

tr '\n' ' ' < input_filename

or remove the newline characters entirely:

tr -d '\n' < input.txt > output.txt

or if you have the GNU version (with its long options)

tr --delete '\n' < input.txt > output.txt

Easy-to-understand Solution

I had this problem. The kicker was that I needed the solution to work on BSD's (Mac OS X) and GNU's (Linux and Cygwin) sed and tr:

$ echo 'foo
bar
baz


foo2
bar2
baz2' \
| tr '\n' '\000' \
| sed 's:\x00\x00.*:\n:g' \
| tr '\000' '\n'

Output:

foo
bar
baz

(has trailing newline)

It works on Linux, OS X, and BSD - even without UTF-8 support or with a crappy terminal.

1. Use tr to swap the newline with another character.

   NULL (\000 or \x00) is nice because it doesn't need UTF-8 support and it's not likely to be used.

2. Use sed to match the NULL

3. Use tr to swap back extra newlines if you need them

The following is a lot simpler than most answers. Also, it is working:

echo `sed -e 's/$/\ |\ /g' file`

This is really simple... I really get irritated when I found the solution. There was just one more back slash missing. This is it:

sed -i "s/\\\\\n//g" filename

If you are unfortunate enough to have to deal with windows line endings you need to remove the \r and the \n

tr '\r\n' ' ' < $input > $output

You can use this method also

sed 'x;G;1!h;s/\n/ /g;$!d'

Explanation

x   - which is used to exchange the data from both space (pattern and hold).
G   - which is used to append the data from hold space to pattern space.
h   - which is used to copy the pattern space to hold space.
1!h - During first line won't copy pattern space to hold space due to \n is
      available in pattern space.
$!d - Clear the pattern space every time before getting next line until the
      last line.

Flow:
When the first line get from the input, exchange is made, so 1 goes to hold space and \n comes to pattern space, then appending the hold space to pattern space, and then substitution is performed and deleted the pattern space.
During the second line exchange is made, 2 goes to hold space and 1 comes to pattern space, then G append the hold space into the pattern space, then h copy the pattern to it and substitution is made and deleted. This operation is continued until eof is reached then print exact result.

Another GNU sed method, almost the same as Zsolt Botykai's answer, but this uses sed's less-frequently used y (transliterate) command, which saves one byte of code (the trailing g):

sed ':a;N;$!ba;y/\n/ /'

One would hope y would run faster than s, (perhaps at tr speeds, 20x faster), but in GNU sed v4.2.2 y is about 4% slower than s.

More portable BSD sed version:

sed -e ':a' -e 'N;$!ba' -e 'y/\n/ /'

I used a hybrid approach to get around the newline thing by using tr to replace newlines with tabs, then replacing tabs with whatever I want. In this case, "
" since I'm trying to generate HTML breaks.

echo -e "a\nb\nc\n" |tr '\n' '\t' | sed 's/\t/ <br> /g'`

You could use xargs — it will replace \n with a space by default.

However, it would have problems if your input has any case of an unterminated quote, e.g. if the quote signs on a given line don't match.

In the sed replacement part, type backslash, hit enter to go to the second line, then end it with /g':

sed 's/>/\
/g'

[root@localhost ~]# echo "1st</first>2nd</second>3rd</third>" | sed 's/>/\
> /g'
1st</first
2nd</second
3rd</third

[root@localhost ~]#

Replace newlines with any string, and replace the last newline too

The pure tr solutions can only replace with a single character, and the pure sed solutions don't replace the last newline of the input. The following solution fixes these problems, and seems to be safe for binary data (even with a UTF-8 locale):

printf '1\n2\n3\n' |
  sed 's/%/%p/g;s/@/%a/g' | tr '\n' @ | sed 's/@/<br>/g;s/%a/@/g;s/%p/%/g'

Result:

1<br>2<br>3<br>

Why didn't I find a simple solution with awk?

awk '{printf $0}' file

printf will print the every line without newlines, if you want to separate the original lines with a space or other:

awk '{printf $0 " "}' file

Finds and replaces using allowing \n

sed -ie -z 's/Marker\n/# Marker Comment\nMarker\n/g' myfile.txt

Marker

Becomes

# Marker Comment

Marker

It is sed that introduces the new-lines after "normal" substitution. First, it trims the new-line char, then it processes according to your instructions, then it introduces a new-line.

Using sed you can replace "the end" of a line (not the new-line char) after being trimmed, with a string of your choice, for each input line; but, sed will output different lines. For example, suppose you wanted to replace the "end of line" with "===" (more general than a replacing with a single space):

PROMPT~$ cat <<EOF |sed 's/$/===/g'
first line
second line
3rd line
EOF

first line===
second line===
3rd line===
PROMPT~$

To replace the new-line char with the string, you can, inefficiently though, use tr , as pointed before, to replace the newline-chars with a "special char" and then use sed to replace that special char with the string you want.

For example:

PROMPT~$ cat <<EOF | tr '\n' $'\x01'|sed -e 's/\x01/===/g'
first line
second line
3rd line
EOF

first line===second line===3rd line===PROMPT~$

sed -i ':a;N;$!ba;s/\n/,/g' test.txt

tr "\n" <file name>

gnu sed has an option -z for null separated records (lines). You can just call:

sed -z 's/\n/ /g'

A shorter awk alternative:

awk 1 ORS=' '

Explanation

An awk program is built up of rules which consist of conditional code-blocks, i.e.:

condition { code-block }

If the code-block is omitted, the default is used: { print $0 }. Thus, the 1 is interpreted as a true condition and print $0 is executed for each line.

When awk reads the input it splits it into records based on the value of RS (Record Separator), which by default is a newline, thus awk will by default parse the input line-wise. The splitting also involves stripping off RS from the input record.

Now, when printing a record, ORS (Output Record Separator) is appended to it, default is again a newline. So by changing ORS to a space all newlines are changed to spaces.

To remove empty lines:

sed -n "s/^$//;t;p;"

Three things.

1. tr (or cat, etc.) is absolutely not needed. (GNU) sed and (GNU) awk, when combined, can do 99.9% of any text processing you need.

2. stream != line based. ed is a line-based editor. sed is not. See sed lecture for more information on the difference. Most people confuse sed to be line-based because it is, by default, not very greedy in its pattern matching for SIMPLE matches - for instance, when doing pattern searching and replacing by one or two characters, it by default only replaces on the first match it finds (unless specified otherwise by the global command). There would not even be a global command if it were line-based rather than STREAM-based, because it would evaluate only lines at a time. Try running ed; you'll notice the difference. ed is pretty useful if you want to iterate over specific lines (such as in a for-loop), but most of the times you'll just want sed.

3. That being said,

   sed -e '{:q;N;s/\n/ /g;t q}' file
   

   works just fine in GNU sed version 4.2.1. The above command will replace all newlines with spaces. It's ugly and a bit cumbersome to type in, but it works just fine. The {}'s can be left out, as they're only included for sanity reasons.

I posted this answer because I have tried with most of the sed commend example provided above which does not work for me in my Unix box and giving me error message Label too long: {:q;N;s/\n/ /g;t q}. Finally I made my requirement and hence shared here which works in all Unix/Linux environment:-

line=$(while read line; do echo -n "$line "; done < yoursourcefile.txt)
echo $line |sed 's/ //g' > sortedoutput.txt

The first line will remove all the new line from file yoursourcefile.txt and will produce a single line. And second sed command will remove all the spaces from it.

Bullet-proof solution. Binary-data-safe and POSIX-compliant, but slow.

POSIX sed requires input according to the POSIX text file and POSIX line definitions, so NULL-bytes and too long lines are not allowed and each line must end with a newline (including the last line). This makes it hard to use sed for processing arbitrary input data.

The following solution avoids sed and instead converts the input bytes to octal codes and then to bytes again, but intercepts octal code 012 (newline) and outputs the replacement string in place of it. As far as I can tell the solution is POSIX-compliant, so it should work on a wide variety of platforms.

od -A n -t o1 -v | tr ' \t' '\n\n' | grep . |
  while read x; do [ "0$x" -eq 012 ] && printf '<br>\n' || printf "\\$x"; done

POSIX reference documentation: sh, shell command language, od, tr, grep, read, [, printf.

Both read, [, and printf are built-ins in at least bash, but that is probably not guaranteed by POSIX, so on some platforms it could be that each input byte will start one or more new processes, which will slow things down. Even in bash this solution only reaches about 50 kB/s, so it's not suited for large files.

Tested on Ubuntu (bash, dash, and busybox), FreeBSD, and OpenBSD.

sed '1h;1!H;$!d
     x;s/\n/ /g' YourFile

This does not work for huge files (buffer limit), but it is very efficient if there is enough memory to hold the file. (Correction H-> 1h;1!H after the good remark of @hilojack )

Another version that change new line while reading (more cpu, less memory)

 sed ':loop
 $! N
 s/\n/ /
 t loop' YourFile

Try this:

echo "a,b"|sed 's/,/\r\n/'

A solution I particularly like is to append all the file in the hold space and replace all newlines at the end of file:

$ (echo foo; echo bar) | sed -n 'H;${x;s/\n//g;p;}'
foobar

However, someone said me the hold space can be finite in some sed implementations.

This might work for you (GNU sed):

sed 'H;$!d;x;:a;s/^((.).*)\2/\1 /;ta;s/.//' file

The H command prepends a newline to the pattern space and then appends the result to the hold space. The normal flow of sed is to remove the following newline from each line, thus this will introduce a newline to the start of the hold space and the replicate the remainder of the file. Once the file has been slurped into the hold space, swap the hold space with the patten space and then use pattern matching to replace all original newlines with spaces. Finally, remove the introduced newline.

This has the advantage of never actually entering a newline string within the sed commands.

The answer with the :a label ...

How can I replace a newline (\n) using sed?

... does not work in freebsd 7.2 on the command line:

( echo foo ; echo bar ) | sed ':a;N;$!ba;s/\n/ /g'
sed: 1: ":a;N;$!ba;s/\n/ /g": unused label 'a;N;$!ba;s/\n/ /g'
foo
bar

But does if you put the sed script in a file or use -e to "build" the sed script...

> (echo foo; echo bar) | sed -e :a -e N -e '$!ba' -e 's/\n/ /g'
foo bar

or ...

> cat > x.sed << eof
:a
N
$!ba
s/\n/ /g
eof

> (echo foo; echo bar) | sed -f x.sed
foo bar

Maybe the sed in OS X is similar.

The Perl version works the way you expected.

perl -i -p -e 's/\n//' file

As pointed out in the comments, it's worth noting that this edits in place. -i.bak will give you a backup of the original file before the replacement in case your regular expression isn't as smart as you thought.

@OP, if you want to replace newlines in a file, you can just use dos2unix (or unix2dox)

dos2unix yourfile yourfile

In some situations maybe you can change RS to some other string or character. This way, \n is available for sub/gsub:

$ gawk 'BEGIN {RS="dn" } {gsub("\n"," ") ;print $0 }' file

The power of shell scripting is that if you do not know how to do it in one way you can do it in another way. And many times you have more things to take into account than make a complex solution on a simple problem.

Regarding the thing that gawk is slow... and reads the file into memory, I do not know this, but to me gawk seems to work with one line at the time and is very very fast (not that fast as some of the others, but the time to write and test also counts).

I process MB and even GB of data, and the only limit I found is line size.

Fast answer

sed ':a;N;$!ba;s/\n/ /g' file

1. :a create a label 'a'
2. N append the next line to the pattern space
3. $! if not the last line, ba branch (go to) label 'a'
4. s substitute, /\n/ regex for new line, / / by a space, /g global match (as many times as it can)

sed will loop through step 1 to 3 until it reach the last line, getting all lines fit in the pattern space where sed will substitute all \n characters

Alternatives

All alternatives, unlike sed will not need to reach the last line to begin the process

with bash, slow

while read line; do printf "%s" "$line "; done < file

with perl, sed-like speed

perl -p -e 's/\n/ /' file

with tr, faster than sed, can replace by one character only

tr '\n' ' ' < file

with paste, tr-like speed, can replace by one character only

paste -s -d ' ' file

with awk, tr-like speed

awk 1 ORS=' ' file

Other alternative like "echo $(< file)" is slow, works only on small files and needs to process the whole file to begin the process.

Long answer from the sed FAQ 5.10

5.10. Why can't I match or delete a newline using the \n escape
sequence? Why can't I match 2 or more lines using \n?

The \n will never match the newline at the end-of-line because the
newline is always stripped off before the line is placed into the
pattern space. To get 2 or more lines into the pattern space, use
the 'N' command or something similar (such as 'H;...;g;').

Sed works like this: sed reads one line at a time, chops off the
terminating newline, puts what is left into the pattern space where
the sed script can address or change it, and when the pattern space
is printed, appends a newline to stdout (or to a file). If the
pattern space is entirely or partially deleted with 'd' or 'D', the
newline is not added in such cases. Thus, scripts like

  sed 's/\n//' file       # to delete newlines from each line             
  sed 's/\n/foo\n/' file  # to add a word to the end of each line         

will NEVER work, because the trailing newline is removed before
the line is put into the pattern space. To perform the above tasks,
use one of these scripts instead:

  tr -d '\n' < file              # use tr to delete newlines              
  sed ':a;N;$!ba;s/\n//g' file   # GNU sed to delete newlines             
  sed 's/$/ foo/' file           # add "foo" to end of each line          

Since versions of sed other than GNU sed have limits to the size of
the pattern buffer, the Unix 'tr' utility is to be preferred here.
If the last line of the file contains a newline, GNU sed will add
that newline to the output but delete all others, whereas tr will
delete all newlines.

To match a block of two or more lines, there are 3 basic choices:
(1) use the 'N' command to add the Next line to the pattern space;
(2) use the 'H' command at least twice to append the current line
to the Hold space, and then retrieve the lines from the hold space
with x, g, or G; or (3) use address ranges (see section 3.3, above)
to match lines between two specified addresses.

Choices (1) and (2) will put an \n into the pattern space, where it
can be addressed as desired ('s/ABC\nXYZ/alphabet/g'). One example
of using 'N' to delete a block of lines appears in section 4.13
("How do I delete a block of specific consecutive lines?"). This
example can be modified by changing the delete command to something
else, like 'p' (print), 'i' (insert), 'c' (change), 'a' (append),
or 's' (substitute).

Choice (3) will not put an \n into the pattern space, but it does
match a block of consecutive lines, so it may be that you don't
even need the \n to find what you're looking for. Since GNU sed
version 3.02.80 now supports this syntax:

  sed '/start/,+4d'  # to delete "start" plus the next 4 lines,           

in addition to the traditional '/from here/,/to there/{...}' range
addresses, it may be possible to avoid the use of \n entirely.

tr '\n' ' ' 

is the command.

Simple and easy to use.

In response to the "tr" solution above, on Windows (probably using the Gnuwin32 version of tr), the proposed solution:

tr '\n' ' ' < input

was not working for me, it'd either error or actually replace the \n w/ '' for some reason.

Using another feature of tr, the "delete" option -d did work though:

tr -d '\n' < input

or '\r\n' instead of '\n'

Yet another option:

tr -s "[:space:]" " " < filename > filename2 && mv filename2 filename

Where tr -s is for:

-s, --squeeze-repeats replace each sequence of a repeated character that is listed in the last specified SET, with a single occurrence of that character

This uses replaces all whitespace sequences in the file with a single space, writes result to a new file, then renames new file back to original name.

I'm not an expert, but I guess in sed you'd first need to append the next line into the pattern space, bij using "N". From the section "Multiline Pattern Space" in "Advanced sed Commands" of the book sed & awk (Dale Dougherty and Arnold Robbins; O'Reilly 1997; page 107 in the preview):

The multiline Next (N) command creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The original contents of pattern space and the new input line are separated by a newline. The embedded newline character can be matched in patterns by the escape sequence "\n". In a multiline pattern space, the metacharacter "^" matches the very first character of the pattern space, and not the character(s) following any embedded newline(s). Similarly, "$" matches only the final newline in the pattern space, and not any embedded newline(s). After the Next command is executed, control is then passed to subsequent commands in the script.

From man sed:

[2addr]N

Append the next line of input to the pattern space, using an embedded newline character to separate the appended material from the original contents. Note that the current line number changes.

I've used this to search (multiple) badly formatted log files, in which the search string may be found on an "orphaned" next line.

On Mac OS X (using FreeBSD sed):

# replace each newline with a space
printf "a\nb\nc\nd\ne\nf" | sed -E -e :a -e '$!N; s/\n/ /g; ta'
printf "a\nb\nc\nd\ne\nf" | sed -E -e :a -e '$!N; s/\n/ /g' -e ta