How to convert DOS Windows newline CRLF to Unix newline LF in a Bash script

Question

How can I programmatically  i e   not using vi  convert DOS Windows newlines to Unix   The dos2unix and unix2dos commands are not available on certain systems  How can I emulate these with commands like sed awk tr

User · Answer

interestingly in my git-bash on windows sed "" did the trick already:

$ echo -e "abc\r" >tst.txt
$ file tst.txt
tst.txt: ASCII text, with CRLF line terminators
$ sed -i "" tst.txt
$ file tst.txt
tst.txt: ASCII text

My guess is that sed ignores them when reading lines from input and always writes unix line endings on output.

User · Answer

For Mac osx if you have homebrew installed   http   brew sh   1   brew install dos2unix  for csv in   csv  do dos2unix -c mac   csv   done    Make sure you have made copies of the files  as this command will modify the files in place  The -c mac option makes the switch to be compatible with osx

User · Answer

This worked for me   tr   r    n   lt  sampledata csv  gt  sampledata2 csv

User · Answer

TIMTOWTDI   perl -pe  s  r n  n   s     n   z  1 n  if eof  PCfile txt   Based on  GordonDavisson  One must consider the possibility of  noeol

User · Answer

If you don t have access to dos2unix  but can read this page  then you can copy paste dos2unix py from here      usr bin env python      convert dos linefeeds  crlf  to unix  lf  usage  dos2unix py  lt input gt   lt output gt      import sys  if len sys argv 1       2    sys exit   doc     content      outsize   0 with open sys argv 1    rb   as infile    content   infile read   with open sys argv 2    wb   as output    for line in content splitlines        outsize    len line    1     output write line     n    print  Done  Saved  s bytes      len content -outsize     Cross-posted from superuser

User · Answer

The solutions posted so far only deal with part of the problem  converting DOS Windows  CRLF into Unix s LF  the part they re missing is that DOS use CRLF as a line separator  while Unix uses LF as a line terminator   The difference is that a DOS file  usually  won t have anything after the last line in the file  while Unix will   To do the conversion properly  you need to add that final LF  unless the file is zero-length  i e  has no lines in it at all    My favorite incantation for this  with a little added logic to handle Mac-style CR-separated files  and not molest files that re already in unix format  is a bit of perl   perl -pe  if   s  r n   n g      f 1    if    f       m     s     n   z  1 n      m 1  PCfile txt   Note that this sends the Unixified version of the file to stdout   If you want to replace the file with a Unixified version  add perl s -i flag

User · Answer

An even simpler awk solution w o a program   awk -v ORS   r n   1  unix txt  gt  dos txt   Technically  1  is your program  b c awk requires one when given option    UPDATE   After revisiting this page for the first time in a long time I realized that no one has yet posted an internal solution  so here is one   while IFS  read -r line  do printf   s n     line    r     done  lt  dos txt  gt  unix txt

User · Answer

Doing this with POSIX is tricky    POSIX Sed does not support  r or  15  Even if it did  the in place option -i is not POSIX POSIX Awk does support  r and  15  however the -i inplace option is not POSIX d2u and dos2unix are not POSIX utilities  but ex is POSIX ex does not support  r   15   n or  12   To remove carriage returns   ex -bsc    awk   sub   r        1   -cx file   To add carriage returns   ex -bsc    awk   sub        r    1   -cx file

User · Answer

You can use vim programmatically with the option -c  command     Dos to Unix   vim file txt -c  set ff unix  -c   wq    Unix to dos   vim file txt -c  set ff dos  -c   wq     set ff unix dos  means change fileformat  ff  of the file to Unix DOS end of line format    wq  means write file to disk and quit the editor  allowing to use the command in a loop

User · Answer

You can use awk  Set the record separator  RS  to a regexp that matches all possible newline character  or characters  And set the output record separator  ORS  to the unix-style newline character   awk  BEGIN RS   r  n  r n  n r  ORS   n   print   windows or macos txt  gt  unix txt

User · Answer

Had just to ponder that same question  on Windows-side  but equally applicable to linux   Suprisingly nobody mentioned a very much automated way of doing CRLF lt - LF conversion for text-files using good old zip -ll option  Info-ZIP    zip -ll textfiles-lf zip files-with-crlf-eol   unzip textfiles-lf zip    NOTE  this would create a zip file preserving the original file names but converting the line endings to LF  Then unzip would extract the files as zip ed  that is with their original names  but with LF-endings   thus prompting to overwrite the local original files if any   Relevant excerpt from the zip --help   zip --help     -l   convert LF to CR LF  -ll CR LF to LF

User · Answer

With bash 4 2 and newer you can use something like this to strip the trailing CR  which only uses bash built-ins   if       str  -1         r      then     str    str   -1   fi

User · Answer

You can use tr to convert from DOS to Unix  however  you can only do this safely if CR appears in your file only as the first byte of a CRLF byte pair   This is usually the case   You then use   tr -d   015   lt DOS-file  gt UNIX-file   Note that the name DOS-file is different from the name UNIX-file  if you try to use the same name twice  you will end up with no data in the file   You can t do it the other way round  with standard  tr     If you know how to enter carriage return into a script  control-V  control-M to enter control-M   then   sed  s  M           DOS to Unix sed  s    M         Unix to DOS   where the   M  is the control-M character   You can also use the bash ANSI-C Quoting mechanism to specify the carriage return   sed   s  r           DOS to Unix sed   s    r         Unix to DOS   However  if you re going to have to do this very often  more than once  roughly speaking   it is far more sensible to install the conversion programs  e g  dos2unix and unix2dos  or perhaps dtou and utod  and use them   If you need to process entire directories and subdirectories  you can use zip   zip -r -ll zipfile zip somedir  unzip zipfile zip   This will create a zip archive with line endings changed from CRLF to CR  unzip will then put the converted files back in place  and ask you file by file - you can answer  Yes-to-all   Credits to  vmsnomad for pointing this out

User · Answer

Using AWK you can do   awk    sub   r         print    dos txt  gt  unix txt   Using Perl you can do   perl -pe  s  r      lt  dos txt  gt  unix txt

User · Answer

Super duper easy with PCRE   As a script  or replace    with your files      usr bin env bash perl -pi -e  s  r n  n g  --         This will overwrite your files in place       I recommend only doing this with a backup  version control or otherwise

User · Answer

I tried sed  s  M     file txt  on OSX as well as several other methods  http   www thingy-ma-jig co uk blog 25-11-2010 fixing-dos-line-endings or http   hintsforums macworld com archive index php t-125 html   None worked  the file remained unchanged  btw Ctrl-v Enter was needed to reproduce  M   In the end I used TextWrangler  Its not strictly command line but it works and it doesn t complain

User · Answer

tr -d   r   lt  file   take a look here for examples using sed     IN UNIX ENVIRONMENT  convert DOS newlines  CR LF  to Unix format  sed  s                       assumes that all lines end with CR LF sed  s  M                    in bash tcsh  press Ctrl-V then Ctrl-M sed  s  x0D                  works on ssed  gsed 3 02 80 or higher    IN UNIX ENVIRONMENT  convert Unix newlines  LF  to DOS format  sed  s    echo -e    r                 command line under ksh sed  s      echo    r                  command line under bash sed  s    echo    r                    command line under zsh sed  s    r                            gsed 3 02 80 or higher   Use sed -i for in-place conversion e g  sed -i  s        file

User · Answer

sed --expression  s  r n  n g    Since the question mentions sed  this is the most straight forward way to use sed to achieve this  What the expression says is replace all carriage-return and line-feed with just line-feed only  That is what you need when you go from Windows to Unix  I verified it works

User · Answer

This problem can be solved with standard tools  but there are sufficiently many traps for the unwary that I recommend you install the flip command  which was written over 20 years ago by Rahul Dhesi  the author of zoo  It does an excellent job converting file formats while  for example  avoiding the inadvertant destruction of binary files  which is a little too easy if you just race around altering every CRLF you see

User · Answer

As an extension to Jonathan Leffler s Unix to DOS solution  to safely convert to DOS when you re unsure of the file s current line endings   sed    M    s    M     This checks that the line does not already end in CRLF before converting to CRLF

User · Answer

On Linux it s easy to convert  M  ctrl-M  to  nix newlines   J  with sed   It will something like this on the CLI  there will actually be a line break in the text   However  the   passes that  J along to sed   sed  s  M    g   lt  ffmpeg log  gt  new log   You get this by using  V  ctrl-V    M  ctrl-M  and    backslash  as you type   sed  s  V M   V J g   lt  ffmpeg log  gt  new log

User · Answer

Just install dos2unix then to convert a file in place use dos2unix  lt filename gt   To output converted text to a different file use dos2unix -n  lt input-file gt   lt output-file gt    You can install it on Ubuntu or Debian with sudo apt install dos2unix  or on macOS using homebrew brew install dos2unix

User · Answer

I made a script based on the accepted answer so you can convert it directly without needing an additional file in the end and removing and renaming afterwards   convert-crlf-to-lf         file   1      tr -d   015   lt   file   gt   file 2     rm -rf   file      mv   file 2   file      just make sure if you have a file like  file1 txt  that  file1 txt2  doesn t already exist or it will be overwritten  I use this as a temporary place to store the file in

[linux] How to convert DOS/Windows newline (CRLF) to Unix newline (LF) in a Bash script?

Examples related to linux

Examples related to windows

Examples related to bash

Examples related to unix

Examples related to newline