[sh] How can I have a newline in a string in sh?

This

STR="Hello\nWorld"
echo $STR

produces as output

Hello\nWorld

instead of

Hello
World

What should I do to have a newline in a string?

Note: This question is not about echo. I'm aware of echo -e, but I'm looking for a solution that allows passing a string (which includes a newline) as an argument to other commands that do not have a similar option to interpret \n's as newlines.

The problem isn't with the shell. The problem is actually with the echo command itself, and the lack of double quotes around the variable interpolation. You can try using echo -e but that isn't supported on all platforms, and one of the reasons printf is now recommended for portability.

You can also try and insert the newline directly into your shell script (if a script is what you're writing) so it looks like...

#!/bin/sh
echo "Hello
World"
#EOF

or equivalently

#!/bin/sh
string="Hello
World"
echo "$string"  # note double quotes!

A $ right before single quotation marks '...\n...' as follows, however double quotation marks doesn't work.

$ echo $'Hello\nWorld'
Hello
World
$ echo $"Hello\nWorld"
Hello\nWorld

What I did based on the other answers was

NEWLINE=$'\n'
my_var="__between eggs and bacon__"
echo "spam${NEWLINE}eggs${my_var}bacon${NEWLINE}knight"

# which outputs:
spam
eggs__between eggs and bacon__bacon
knight

This isn't ideal, but I had written a lot of code and defined strings in a way similar to the method used in the question. The accepted solution required me to refactor a lot of the code so instead, I replaced every \n with "$'\n'" and this worked for me.

Those picky ones that need just the newline and despise the multiline code that breaks indentation, could do:

IFS="$(printf '\nx')"
IFS="${IFS%x}"

Bash (and likely other shells) gobble all the trailing newlines after command substitution, so you need to end the printf string with a non-newline character and delete it afterwards. This can also easily become a oneliner.

IFS="$(printf '\nx')" IFS="${IFS%x}"

I know this is two actions instead of one, but my indentation and portability OCD is at peace now :) I originally developed this to be able to split newline-only separated output and I ended up using a modification that uses \r as the terminating character. That makes the newline splitting work even for the dos output ending with \r\n.

IFS="$(printf '\n\r')"

1. The only simple alternative is to actually type a new line in the variable:

   $ STR='new
   line'
   $ printf '%s' "$STR"
   new
   line
   

   Yes, that means writing Enter where needed in the code.

2. There are several equivalents to a new line character.

   \n           ### A common way to represent a new line character.
   \012         ### Octal value of a new line character.
   \x0A         ### Hexadecimal value of a new line character.
   

   But all those require "an interpretation" by some tool (POSIX printf):

   echo -e "new\nline"           ### on POSIX echo, `-e` is not required.
   printf 'new\nline'            ### Understood by POSIX printf.
   printf 'new\012line'          ### Valid in POSIX printf.
   printf 'new\x0Aline'       
   printf '%b' 'new\0012line'    ### Valid in POSIX printf.
   

   And therefore, the tool is required to build a string with a new-line:

   $ STR="$(printf 'new\nline')"
   $ printf '%s' "$STR"
   new
   line
   

3. In some shells, the sequence $' is an special shell expansion. Known to work in ksh93, bash and zsh:

   $ STR=$'new\nline'
   

4. Of course, more complex solutions are also possible:

   $ echo '6e65770a6c696e650a' | xxd -p -r
   new
   line
   

   Or

   $ echo "new line" | sed 's/ \+/\n/g'
   new
   line
   

I'm no bash expert, but this one worked for me:

STR1="Hello"
STR2="World"
NEWSTR=$(cat << EOF
$STR1

$STR2
EOF
)
echo "$NEWSTR"

I found this easier to formatting the texts.

I wasn't really happy with any of the options here. This is what worked for me.

str=$(printf "%s" "first line")
str=$(printf "$str\n%s" "another line")
str=$(printf "$str\n%s" "and another line")

Disclaimer: I first wrote this and then stumbled upon this question. I thought this solution wasn't yet posted, and saw that tlwhitec did post a similar answer. Still I'm posting this because I hope it's a useful and thorough explanation.

Short answer:

This seems quite a portable solution, as it works on quite some shells (see comment).
This way you can get a real newline into a variable.

The benefit of this solution is that you don't have to use newlines in your source code, so you can indent your code any way you want, and the solution still works. This makes it robust. It's also portable.

# Robust way to put a real newline in a variable (bash, dash, ksh, zsh; indentation-resistant).
nl="$(printf '\nq')"
nl=${nl%q}

Longer answer:

Explanation of the above solution:

The newline would normally lost due to command substitution, but to prevent that, we add a 'q' and remove it afterwards. (The reason for the double quotes is explained further below.)

We can prove that the variable contains an actual newline character (0x0A):

printf '%s' "$nl" | hexdump -C
00000000  0a  |.|
00000001

(Note that the '%s' was needed, otherwise printf will translate a literal '\n' string into an actual 0x0A character, meaning we would prove nothing.)

Of course, instead of the solution proposed in this answer, one could use this as well (but...):

nl='
'

... but that's less robust and can be easily damaged by accidentally indenting the code, or by forgetting to dedent it afterwards, which makes it inconvenient to use in (indented) functions, whereas the earlier solution is robust.

Now, as for the double quotes:
The reason for the double quotes " surrounding the command substitution as in nl="$(printf '\nq')" is that you can then even prefix the variable assignment with the local keyword or builtin (such as in functions), and it will still work on all shells, whereas otherwise the dash shell would have trouble, in the sense that dash would otherwise lose the 'q' and you'd end up with an empty 'nl' variable (again, due to command substitution).
That issue is better illustrated with another example:

dash_trouble_example() {
    e=$(echo hello world) # Not using 'local'.
    echo "$e" # Fine. Outputs 'hello world' in all shells.

    local e=$(echo hello world) # But now, when using 'local' without double quotes ...:
    echo "$e" # ... oops, outputs just 'hello' in dash,
              # ... but 'hello world' in bash and zsh.

    local f="$(echo hello world)" # Finally, using 'local' and surrounding with double quotes.
    echo "$f" # Solved. Outputs 'hello world' in dash, zsh, and bash.

    # So back to our newline example, if we want to use 'local', we need
    # double quotes to surround the command substitution:
    # (If we didn't use double quotes here, then in dash the 'nl' variable
    # would be empty.)
    local nl="$(printf '\nq')"
    nl=${nl%q}
}

Practical example of the above solution:

# Parsing lines in a for loop by setting IFS to a real newline character:

nl="$(printf '\nq')"
nl=${nl%q}

IFS=$nl

for i in $(printf '%b' 'this is line 1\nthis is line 2'); do
    echo "i=$i"
done

# Desired output:
# i=this is line 1
# i=this is line 2

# Exercise:
# Try running this example without the IFS=$nl assignment, and predict the outcome.

Echo is so nineties and so fraught with perils that its use should result in core dumps no less than 4GB. Seriously, echo's problems were the reason why the Unix Standardization process finally invented the printf utility, doing away with all the problems.

So to get a newline in a string:

FOO="hello
world"
BAR=$(printf "hello\nworld\n") # Alternative; note: final newline is deleted
printf '<%s>\n' "$FOO"
printf '<%s>\n' "$BAR"

There! No SYSV vs BSD echo madness, everything gets neatly printed and fully portable support for C escape sequences. Everybody please use printf now and never look back.

I find the -e flag elegant and straight forward

bash$ STR="Hello\nWorld"

bash$ echo -e $STR
Hello
World

If the string is the output of another command, I just use quotes

indexes_diff=$(git diff index.yaml)
echo "$indexes_diff"

On my system (Ubuntu 17.10) your example just works as desired, both when typed from the command line (into sh) and when executed as a sh script:

[bash]§ sh
$ STR="Hello\nWorld"
$ echo $STR
Hello
World
$ exit
[bash]§ echo "STR=\"Hello\nWorld\"
> echo \$STR" > test-str.sh
[bash]§ cat test-str.sh 
STR="Hello\nWorld"
echo $STR
[bash]§ sh test-str.sh 
Hello
World

I guess this answers your question: it just works. (I have not tried to figure out details such as at what moment exactly the substitution of the newline character for \n happens in sh).

However, i noticed that this same script would behave differently when executed with bash and would print out Hello\nWorld instead:

[bash]§ bash test-str.sh
Hello\nWorld

I've managed to get the desired output with bash as follows:

[bash]§ STR="Hello
> World"
[bash]§ echo "$STR"

Note the double quotes around $STR. This behaves identically if saved and run as a bash script.

The following also gives the desired output:

[bash]§ echo "Hello
> World"