[macos] OS X Terminal UTF-8 issues

Okay, so I finally got myself a MacBook Air after 15 years of linux. And before I got it my big concern was UTF-8 support because no matter if I get files sent to me from windows or mac-clients theres always issues with encoding, while on ubuntu I can be sure that all output no matter what program will produce perfect utf-8 encoded data.

And now on my second day (today) with OS X Im tearing my hair of by frustration. Why?

When I open Nano and type some swedish characters like ÅÄÖ in it, it puts out blank characters at the end of the line (which i guess is the other byte in each character)

When I open python and try using swedish characters, it does not output anything at all

When I connect to a Ubuntu server trough SSH I cant type åäö in bash, tough it works in VIM (still trough SSH). And in nano backspace does not work, but if check the box "Delete sends ctrl+H" in the Terminal preferences, backspace starts working in nano but stops working in VIM.

I've tried unchecking all other encodings then UTF-8 in terminal preferences but that does not seem to work either.

I'm sure that every non US-person must have the same issues, so hove do I fix them? I just want full UTF-8 support... :'(

Check whether nano was actually built with UTF-8 support, using nano --version. Here it is on Cygwin:

nano --version
 GNU nano version 2.2.5 (compiled 21:04:20, Nov  3 2010)
 (C) 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007,
 2008, 2009 Free Software Foundation, Inc.
 Email: [email protected]    Web: http://www.nano-editor.org/
 Compiled options: --enable-color --enable-extra --enable-multibuffer
 --enable-nanorc --enable-utf8

Note the last bit.

For me, this helped: I checked locale on my local shell in terminal

$ locale
LANG="cs_CZ.UTF-8"
LC_COLLATE="cs_CZ.UTF-8"

Then connected to any remote host I am using via ssh and edited file /etc/profile as root - at the end I added line:

export LANG=cs_CZ.UTF-8

After next connection it works fine in bash, ls and nano.

The following is a summary of what you need to do under OS X Mavericks (10.9). This is all summarized in

http://hints.macworld.com/article.php?story=20060825071728278

1. Go to Terminal->Preferences->Settings->Advanced.

   Under International, make sure the character encoding is set to Unicode (UTF-8).

   Also, and this is key: under Emulation, make sure that Escape non-ASCII input with Control-V is unchecked (i.e. is not set).

   These two settings fix things for Terminal.

2. Make sure your locale is set to something that ends in .UTF-8. Type locale and look at the LC_CTYPE line. If it doesn't say something like en_US.UTF-8 (the stuff before the dot might change if you are using a non-US-English locale), then in your Bash .profile or .bashrc in your home directory, add a line like this:

   export LC_CTYPE=en_US.UTF-8
   

   This will fix things for command-line programs in general.

3. Add the following lines to .inputrc in your home directory (create it if necessary):

   set meta-flag on
   set input-meta on
   set output-meta on
   set convert-meta off
   

   This makes Bash be eight-bit clean, so it will pass UTF-8 characters in and out without messing with them.

Keep in mind you will have to restart Bash (e.g. close and reopen the Terminal window) to get it to pay attention to all the settings you make in 2 and 3 above.

To make nano work as you want it to, try:

export LANG="UTF-8"

Or get a newer version of nano via MacPorts:

# cf. http://www.macports.org/install.php
port info nano
port variants nano
sudo port install nano +utf8 +color +no_wrap

With respect to ssh & UTF-8 issues comment out SendEnv LANG LC_* in /etc/ssh_config.

See: Terminal in OS X Lion: can't write åäö on remote machine

Go to Terminal -> Preferences -> Advanced (Tab) go down to International and select Unicode (UTF-8) as Character Encoding.

And tick Set locale environment variables on startup.

Try

1. Having a Powerline compatible font installed https://github.com/powerline/fonts
2. Setting these ENV vars in .zshrc or .bashrc:

LANG="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_ALL="en_US.UTF-8"

Since nano is a terminal application. I guess it's more a terminal problem than a nano problem.

I met similar problems at OS X (I cannot input and view the Chinese characters at terminal).

I tried tweaking the system setting through OS X UI whose real effect is change the environment variable LANG.

So finally I just add some stuff into the ~/.bashrc to fix the problem.

# I'm Chinese and I prefer English manual
export LC_COLLATE="zh_CN.UTF-8"
export LC_CTYPE="zh_CN.UTF-8"
export LC_MESSAGES="en_US.UTF-8"
export LC_MONETARY="zh_CN.UTF-8"
export LC_NUMERIC="zh_CN.UTF-8"
export LC_TIME="zh_CN.UTF-8"

BTW, don't set LC_ALL which will override all the other LC_* settings.

My terminal was just acting silly, not printing out åäö. I found (and set) this setting:

Under Terminal -> Preferences... -> Profiles -> Advanced.

Seems to have fixed my problem.

In my case, simply using the uxterm command instead of xterm solved the problem. It's available in /opt/X11/bin/uxterm by installing the XQuartz package provided by Apple.

Unfortunately, the Preferences dialog is not always very helpful, but by tweaking around you should be able to get everything working.

To be able to type Swedish characters in Terminal, add the following lines to your ~/.inputrc (most likely you must create this file):

set input-meta on
set output-meta on
set convert-meta off

This should do the work both with utf8 and other codings in bash, nano and many other programs. Some programs, like tmux, also depends on the locale. Then, adding for instance export LC_ALL=en_US.UTF-8 to your ~/.profile file should help, but keep in mind that a few (mainly obscure) programs require a standard locale, so if you have trouble running or compiling a program, try going back to LC_ALL=C.

Some references that may be helpful:

• http://homepage.mac.com/thgewecke/mlingos9.html#unicode
• http://hints.macworld.com/article.php?story=20060825071728278

Short versatile answer (fits to other national languages, even Lithuanian or Russian)

• open Terminal
• edit .profile in home directory - nano .profile or in Catalina or newer nano .zshenv
• add line export LC_ALL=en_US.UTF-8
• press Ctrl+x and Y (exit and save)

This solved for me even small country rare national characters. You may need to close and open Terminal to make changes effective.

Also if you like Linux behavior (use lot of Alt shortcuts like Alt+. or Alt+, in mc) then you should disable Mac style Option key function: Terminal->Preferences->Profiles->Keyboard and check box: Use Option as Meta key