UTF-8 all the way through

Question

I m setting up a new server and want to support UTF-8 fully in my web application  I have tried this in the past on existing servers and always seem to end up having to fall back to ISO-8859-1   Where exactly do I need to set the encoding charsets  I m aware that I need to configure Apache  MySQL  and PHP to do this     is there some standard checklist I can follow  or perhaps troubleshoot where the mismatches occur   This is for a new Linux server  running MySQL 5  PHP  5 and Apache 2

User · Answer

In my case  I was using mb split  which uses regex   Therefore I also had to manually make sure the regex encoding was utf-8 by doing mb regex encoding  UTF-8     As a side note  I also discovered by running mb internal encoding   that the internal encoding wasn t utf-8  and I changed that by running mb internal encoding  UTF-8

User · Answer

In PHP  you ll need to either use the multibyte functions  or turn on mbstring func overload  That way things like strlen will work if you have characters that take more than one byte   You ll also need to identify the character set of your responses  You can either use AddDefaultCharset  as above  or write PHP code that returns the header   Or you can add a META tag to your HTML documents

User · Answer

I d like to add one thing to chazomaticus  excellent answer   Don t forget the META tag either  like this  or the HTML4 or XHTML version of it     lt meta charset  utf-8  gt    That seems trivial  but IE7 has given me problems with that before   I was doing everything right  the database  database connection and Content-Type HTTP header were all set to UTF-8  and it worked fine in all other browsers  but Internet Explorer still insisted on using the  Western European  encoding   It turned out the page was missing the META tag  Adding that solved the problem   Edit   The W3C actually has a rather large section dedicated to I18N  They have a number of articles related to this issue  ndash  describing the HTTP   X HTML and CSS side of things    FAQ  Changing  X HTML page encoding to UTF-8 Declaring character encodings in HTML Tutorial  Character sets  amp  encodings in XHTML  HTML and CSS Setting the HTTP charset parameter   They recommend using both the HTTP header and HTML meta tag  or XML declaration in case of XHTML served as XML

User · Answer

In addition to setting default charset in php ini  you can send the correct charset using header   from within your code  before any output   header  Content-Type  text html  charset utf-8      Working with Unicode in PHP is easy as long as you realize that most of the string functions don t work with Unicode  and some might mangle strings completely  PHP considers  characters  to be 1 byte long  Sometimes this is okay  for example  explode   only looks for a byte sequence and uses it as a separator -- so it doesn t matter what actual characters you look for   But other times  when the function is actually designed to work on characters  PHP has no idea that your text has multi-byte characters that are found with Unicode   A good library to check into is phputf8  This rewrites all of the  bad  functions so you can safely work on UTF8 strings  There are extensions like the mbstring extension that try to do this for you  too  but I prefer using the library because it s more portable  but I write mass-market products  so that s important for me   But phputf8 can use mbstring behind the scenes  anyway  to increase performance

User · Answer

In addition to setting default charset in php ini  you can send the correct charset using header   from within your code  before any output   header  Content-Type  text html  charset utf-8      Working with Unicode in PHP is easy as long as you realize that most of the string functions don t work with Unicode  and some might mangle strings completely  PHP considers  characters  to be 1 byte long  Sometimes this is okay  for example  explode   only looks for a byte sequence and uses it as a separator -- so it doesn t matter what actual characters you look for   But other times  when the function is actually designed to work on characters  PHP has no idea that your text has multi-byte characters that are found with Unicode   A good library to check into is phputf8  This rewrites all of the  bad  functions so you can safely work on UTF8 strings  There are extensions like the mbstring extension that try to do this for you  too  but I prefer using the library because it s more portable  but I write mass-market products  so that s important for me   But phputf8 can use mbstring behind the scenes  anyway  to increase performance

User · Answer

First of all if you are in  lt  5 3PHP then no  You ve got a ton of problems to tackle    I am surprised that none has mentioned the intl library  the one that has good support for unicode  graphemes  string operations   localisation and many more  see below    I will quote some information about unicode support in PHP by Elizabeth Smith s slides at PHPBenelux 14  INTL  Good    Wrapper around ICU library Standardised locales  set locale per script Number formatting Currency formatting Message formatting  replaces gettext  Calendars  dates  timezone and time Transliterator Spoofchecker Resource bundles Convertors IDN support Graphemes Collation Iterators   Bad    Does not support zend multibite Does not support HTTP input output conversion Does not support function overloading   mb string   Enables zend multibyte support Supports transparent HTTP in out encoding Provides some wrappers for funtionallity such as strtoupper   ICONV   Primary for charset conversion Output buffer handler mime encoding functionality conversion some string helpers  len  substr  strpos  strrpos  Stream Filter stream filter append  fp   convert iconv ISO-2022-JP EUC-JP     DATABASES   mysql  Charset and collation on tables and on connection  not the collation   Also don t use mysql - msqli or PDO postgresql  pg set client encoding sqlite 3   Make sure it was compiled with unicode and intl support   Some other Gotchas   You cannot use unicode filenames with PHP and windows unless you use a 3rd part extension   Send everything in ASCII if you are using exec  proc open and other command line calls Plain text is not plain text  files have encodings You can convert files on the fly with the iconv filter   I ll update this answer in case things change features added and so on

User · Answer

The only thing I would add to these amazing answers is to emphasize on saving your files in utf8 encoding  i have noticed that browsers accept this property over setting utf8 as your code encoding  Any decent text editor will show you this  for example Notepad   has a menu option for file enconding  it shows you the current encoding and enables you to change it  For all my php files I use utf8 without BOM    Sometime ago i had someone ask me to add utf8 support for a php mysql application designed by someone else  i noticed that all files were encoded in ANSI  so I had to use ICONV to convert all files  change the database tables to use the utf8 charset and utf8 general ci collate  add  SET NAMES utf8  to the database abstraction layer after the connection  if using 5 3 6 or earlier otherwise you have to use charset utf8 in the connection string  and change string functions to use the php multibyte string functions equivalent

User · Answer

if you want a mysql solution   I had similar issues with 2 of my projects  after a server migration  After searching and trying a lot of solutions i came across with this one  nothing before this one worked    mysqli set charset  con  utf8      After adding  this line to my config file everything works fine   I found this solution https   www w3schools com PHP func mysqli set charset asp when i was looking to solve a insert from html query  good luck

User · Answer

I recently discovered that using strtolower   can cause issues where the data is truncated after a special character   The solution was to use   mb strtolower  string   UTF-8         mb  uses MultiByte  It supports more characters but in general is a little slower

User · Answer

Data Storage   Specify the utf8mb4 character set on all tables and text columns in your database   This makes MySQL physically store and retrieve values encoded natively in UTF-8   Note that MySQL will implicitly use utf8mb4 encoding if a utf8mb4   collation is specified  without any explicit character set    In older versions of MySQL   lt  5 5 3   you ll unfortunately be forced to use simply utf8  which only supports a subset of Unicode characters   I wish I were kidding    Data Access   In your application code  e g  PHP   in whatever DB access method you use  you ll need to set the connection charset to utf8mb4   This way  MySQL does no conversion from its native UTF-8 when it hands data off to your application and vice versa   Some drivers provide their own mechanism for configuring the connection character set  which both updates its own internal state and informs MySQL of the encoding to be used on the connection   this is usually the preferred approach    In PHP   If you re using the PDO abstraction layer with PHP   5 3 6  you can specify charset in the DSN    dbh   new PDO  mysql charset utf8mb4      If you re using mysqli  you can call set charset       mysqli- gt set charset  utf8mb4             object oriented style   mysqli set charset  link   utf8mb4        procedural style   If you re stuck with plain mysql but happen to be running PHP   5 2 3  you can call mysql set charset     If the driver does not provide its own mechanism for setting the connection character set  you may have to issue a query to tell MySQL how your application expects data on the connection to be encoded  SET NAMES  utf8mb4    The same consideration regarding utf8mb4 utf8 applies as above    Output   If your application transmits text to other systems  they will also need to be informed of the character encoding   With web applications  the browser must be informed of the encoding in which data is sent  through HTTP response headers or HTML metadata    In PHP  you can use the default charset php ini option  or manually issue the Content-Type MIME header yourself  which is just more work but has the same effect   When encoding the output using json encode    add JSON UNESCAPED UNICODE as a second parameter    Input   Unfortunately  you should verify every received string as being valid UTF-8 before you try to store it or use it anywhere   PHP s mb check encoding   does the trick  but you have to use it religiously   There s really no way around this  as malicious clients can submit data in whatever encoding they want  and I haven t found a trick to get PHP to do this for you reliably   From my reading of the current HTML spec  the following sub-bullets are not necessary or even valid anymore for modern HTML   My understanding is that browsers will work with and submit data in the character set specified for the document   However  if you re targeting older versions of HTML  XHTML  HTML4  etc    these points may still be useful   For HTML before HTML5 only  you want all data sent to you by browsers to be in UTF-8   Unfortunately  if you go by the only way to reliably do this is add the accept-charset attribute to all your  lt form gt  tags   lt form     accept-charset  quot UTF-8 quot  gt   For HTML before HTML5 only  note that the W3C HTML spec says that clients  quot should quot  default to sending forms back to the server in whatever charset the server served  but this is apparently only a recommendation  hence the need for being explicit on every single  lt form gt  tag     Other Code Considerations   Obviously enough  all files you ll be serving  PHP  HTML  JavaScript  etc   should be encoded in valid UTF-8   You need to make sure that every time you process a UTF-8 string  you do so safely   This is  unfortunately  the hard part   You ll probably want to make extensive use of PHP s mbstring extension   PHP s built-in string operations are not by default UTF-8 safe   There are some things you can safely do with normal PHP string operations  like concatenation   but for most things you should use the equivalent mbstring function   To know what you re doing  read  not mess it up   you really need to know UTF-8 and how it works on the lowest possible level   Check out any of the links from utf8 com for some good resources to learn everything you need to know

User · Answer

In addition to setting default charset in php ini  you can send the correct charset using header   from within your code  before any output   header  Content-Type  text html  charset utf-8      Working with Unicode in PHP is easy as long as you realize that most of the string functions don t work with Unicode  and some might mangle strings completely  PHP considers  characters  to be 1 byte long  Sometimes this is okay  for example  explode   only looks for a byte sequence and uses it as a separator -- so it doesn t matter what actual characters you look for   But other times  when the function is actually designed to work on characters  PHP has no idea that your text has multi-byte characters that are found with Unicode   A good library to check into is phputf8  This rewrites all of the  bad  functions so you can safely work on UTF8 strings  There are extensions like the mbstring extension that try to do this for you  too  but I prefer using the library because it s more portable  but I write mass-market products  so that s important for me   But phputf8 can use mbstring behind the scenes  anyway  to increase performance

User · Answer

in connection php  mysqli set charset  con    utf8      and in sql collation utf 8

User · Answer

If you want MySQL server to decide character set  and not PHP as a client  old behaviour  preferred  in my opinion   try adding skip-character-set-client-handshake to your my cnf  under  mysqld   and restart mysql   This may cause troubles in case you re using anything other than UTF8

User · Answer

Data Storage   Specify the utf8mb4 character set on all tables and text columns in your database   This makes MySQL physically store and retrieve values encoded natively in UTF-8   Note that MySQL will implicitly use utf8mb4 encoding if a utf8mb4   collation is specified  without any explicit character set    In older versions of MySQL   lt  5 5 3   you ll unfortunately be forced to use simply utf8  which only supports a subset of Unicode characters   I wish I were kidding    Data Access   In your application code  e g  PHP   in whatever DB access method you use  you ll need to set the connection charset to utf8mb4   This way  MySQL does no conversion from its native UTF-8 when it hands data off to your application and vice versa   Some drivers provide their own mechanism for configuring the connection character set  which both updates its own internal state and informs MySQL of the encoding to be used on the connection   this is usually the preferred approach    In PHP   If you re using the PDO abstraction layer with PHP   5 3 6  you can specify charset in the DSN    dbh   new PDO  mysql charset utf8mb4      If you re using mysqli  you can call set charset       mysqli- gt set charset  utf8mb4             object oriented style   mysqli set charset  link   utf8mb4        procedural style   If you re stuck with plain mysql but happen to be running PHP   5 2 3  you can call mysql set charset     If the driver does not provide its own mechanism for setting the connection character set  you may have to issue a query to tell MySQL how your application expects data on the connection to be encoded  SET NAMES  utf8mb4    The same consideration regarding utf8mb4 utf8 applies as above    Output   If your application transmits text to other systems  they will also need to be informed of the character encoding   With web applications  the browser must be informed of the encoding in which data is sent  through HTTP response headers or HTML metadata    In PHP  you can use the default charset php ini option  or manually issue the Content-Type MIME header yourself  which is just more work but has the same effect   When encoding the output using json encode    add JSON UNESCAPED UNICODE as a second parameter    Input   Unfortunately  you should verify every received string as being valid UTF-8 before you try to store it or use it anywhere   PHP s mb check encoding   does the trick  but you have to use it religiously   There s really no way around this  as malicious clients can submit data in whatever encoding they want  and I haven t found a trick to get PHP to do this for you reliably   From my reading of the current HTML spec  the following sub-bullets are not necessary or even valid anymore for modern HTML   My understanding is that browsers will work with and submit data in the character set specified for the document   However  if you re targeting older versions of HTML  XHTML  HTML4  etc    these points may still be useful   For HTML before HTML5 only  you want all data sent to you by browsers to be in UTF-8   Unfortunately  if you go by the only way to reliably do this is add the accept-charset attribute to all your  lt form gt  tags   lt form     accept-charset  quot UTF-8 quot  gt   For HTML before HTML5 only  note that the W3C HTML spec says that clients  quot should quot  default to sending forms back to the server in whatever charset the server served  but this is apparently only a recommendation  hence the need for being explicit on every single  lt form gt  tag     Other Code Considerations   Obviously enough  all files you ll be serving  PHP  HTML  JavaScript  etc   should be encoded in valid UTF-8   You need to make sure that every time you process a UTF-8 string  you do so safely   This is  unfortunately  the hard part   You ll probably want to make extensive use of PHP s mbstring extension   PHP s built-in string operations are not by default UTF-8 safe   There are some things you can safely do with normal PHP string operations  like concatenation   but for most things you should use the equivalent mbstring function   To know what you re doing  read  not mess it up   you really need to know UTF-8 and how it works on the lowest possible level   Check out any of the links from utf8 com for some good resources to learn everything you need to know

User · Answer

Data Storage   Specify the utf8mb4 character set on all tables and text columns in your database   This makes MySQL physically store and retrieve values encoded natively in UTF-8   Note that MySQL will implicitly use utf8mb4 encoding if a utf8mb4   collation is specified  without any explicit character set    In older versions of MySQL   lt  5 5 3   you ll unfortunately be forced to use simply utf8  which only supports a subset of Unicode characters   I wish I were kidding    Data Access   In your application code  e g  PHP   in whatever DB access method you use  you ll need to set the connection charset to utf8mb4   This way  MySQL does no conversion from its native UTF-8 when it hands data off to your application and vice versa   Some drivers provide their own mechanism for configuring the connection character set  which both updates its own internal state and informs MySQL of the encoding to be used on the connection   this is usually the preferred approach    In PHP   If you re using the PDO abstraction layer with PHP   5 3 6  you can specify charset in the DSN    dbh   new PDO  mysql charset utf8mb4      If you re using mysqli  you can call set charset       mysqli- gt set charset  utf8mb4             object oriented style   mysqli set charset  link   utf8mb4        procedural style   If you re stuck with plain mysql but happen to be running PHP   5 2 3  you can call mysql set charset     If the driver does not provide its own mechanism for setting the connection character set  you may have to issue a query to tell MySQL how your application expects data on the connection to be encoded  SET NAMES  utf8mb4    The same consideration regarding utf8mb4 utf8 applies as above    Output   If your application transmits text to other systems  they will also need to be informed of the character encoding   With web applications  the browser must be informed of the encoding in which data is sent  through HTTP response headers or HTML metadata    In PHP  you can use the default charset php ini option  or manually issue the Content-Type MIME header yourself  which is just more work but has the same effect   When encoding the output using json encode    add JSON UNESCAPED UNICODE as a second parameter    Input   Unfortunately  you should verify every received string as being valid UTF-8 before you try to store it or use it anywhere   PHP s mb check encoding   does the trick  but you have to use it religiously   There s really no way around this  as malicious clients can submit data in whatever encoding they want  and I haven t found a trick to get PHP to do this for you reliably   From my reading of the current HTML spec  the following sub-bullets are not necessary or even valid anymore for modern HTML   My understanding is that browsers will work with and submit data in the character set specified for the document   However  if you re targeting older versions of HTML  XHTML  HTML4  etc    these points may still be useful   For HTML before HTML5 only  you want all data sent to you by browsers to be in UTF-8   Unfortunately  if you go by the only way to reliably do this is add the accept-charset attribute to all your  lt form gt  tags   lt form     accept-charset  quot UTF-8 quot  gt   For HTML before HTML5 only  note that the W3C HTML spec says that clients  quot should quot  default to sending forms back to the server in whatever charset the server served  but this is apparently only a recommendation  hence the need for being explicit on every single  lt form gt  tag     Other Code Considerations   Obviously enough  all files you ll be serving  PHP  HTML  JavaScript  etc   should be encoded in valid UTF-8   You need to make sure that every time you process a UTF-8 string  you do so safely   This is  unfortunately  the hard part   You ll probably want to make extensive use of PHP s mbstring extension   PHP s built-in string operations are not by default UTF-8 safe   There are some things you can safely do with normal PHP string operations  like concatenation   but for most things you should use the equivalent mbstring function   To know what you re doing  read  not mess it up   you really need to know UTF-8 and how it works on the lowest possible level   Check out any of the links from utf8 com for some good resources to learn everything you need to know

User · Answer

Unicode support in PHP is still a huge mess  While it s capable of converting an ISO8859 string  which it uses internally  to utf8  it lacks the capability to work with unicode strings natively  which means all the string processing functions will mangle and corrupt your strings  So you have to either use a separate library for proper utf8 support  or rewrite all the string handling functions yourself   The easy part is just specifying the charset in HTTP headers and in the database and such  but none of that matters if your PHP code doesn t output valid UTF8  That s the hard part  and PHP gives you virtually no help there   I think PHP6 is supposed to fix the worst of this  but that s still a while away

User · Answer

In PHP  you ll need to either use the multibyte functions  or turn on mbstring func overload  That way things like strlen will work if you have characters that take more than one byte   You ll also need to identify the character set of your responses  You can either use AddDefaultCharset  as above  or write PHP code that returns the header   Or you can add a META tag to your HTML documents

User · Answer

Just a note   You are facing the problem of your non-latin characters is showing as             you asked a question  and it got closed with a reference to this canonical question  you tried everything and no matter what you do you still get            from MySQL   That is mostly because you are testing on your old data which has been inserted to the database using the wrong charset and got converted and stored to actually the question mark characters    Which means you lost your original text forever and no matter what you try you will get           re applying what you have learned from the answers of this question on a fresh data could solve your problem

User · Answer

In addition to setting default charset in php ini  you can send the correct charset using header   from within your code  before any output   header  Content-Type  text html  charset utf-8      Working with Unicode in PHP is easy as long as you realize that most of the string functions don t work with Unicode  and some might mangle strings completely  PHP considers  characters  to be 1 byte long  Sometimes this is okay  for example  explode   only looks for a byte sequence and uses it as a separator -- so it doesn t matter what actual characters you look for   But other times  when the function is actually designed to work on characters  PHP has no idea that your text has multi-byte characters that are found with Unicode   A good library to check into is phputf8  This rewrites all of the  bad  functions so you can safely work on UTF8 strings  There are extensions like the mbstring extension that try to do this for you  too  but I prefer using the library because it s more portable  but I write mass-market products  so that s important for me   But phputf8 can use mbstring behind the scenes  anyway  to increase performance

User · Answer

I d like to add one thing to chazomaticus  excellent answer   Don t forget the META tag either  like this  or the HTML4 or XHTML version of it     lt meta charset  utf-8  gt    That seems trivial  but IE7 has given me problems with that before   I was doing everything right  the database  database connection and Content-Type HTTP header were all set to UTF-8  and it worked fine in all other browsers  but Internet Explorer still insisted on using the  Western European  encoding   It turned out the page was missing the META tag  Adding that solved the problem   Edit   The W3C actually has a rather large section dedicated to I18N  They have a number of articles related to this issue  ndash  describing the HTTP   X HTML and CSS side of things    FAQ  Changing  X HTML page encoding to UTF-8 Declaring character encodings in HTML Tutorial  Character sets  amp  encodings in XHTML  HTML and CSS Setting the HTTP charset parameter   They recommend using both the HTTP header and HTML meta tag  or XML declaration in case of XHTML served as XML

User · Answer

I recently discovered that using strtolower   can cause issues where the data is truncated after a special character   The solution was to use   mb strtolower  string   UTF-8         mb  uses MultiByte  It supports more characters but in general is a little slower

User · Answer

The top answer is excellent  Here is what I had to on a regular debian php mysql setup      storage    debian  apparently already utf-8     retrieval    the mysql database was stored in utf-8      but apparently php was requesting iso  this worked         notice  utf8   without dash  this is a mysql encoding    mysql set charset  utf8        delivery    php ini did not have a default charset       it was commented out  shared host  and    no http encoding was specified in the apache headers     this made apache send out a utf-8 header     and perhaps made php actually send out utf-8        notice  utf-8   with dash  this is a php encoding    ini set  default charset   utf-8        submission    this worked in all major browsers once apache    was sending out the utf-8 header  i didnt add    the accept-charset attribute      processing    changed a few commands in php  like substr     to mb substr   that was all

User · Answer

Data Storage   Specify the utf8mb4 character set on all tables and text columns in your database   This makes MySQL physically store and retrieve values encoded natively in UTF-8   Note that MySQL will implicitly use utf8mb4 encoding if a utf8mb4   collation is specified  without any explicit character set    In older versions of MySQL   lt  5 5 3   you ll unfortunately be forced to use simply utf8  which only supports a subset of Unicode characters   I wish I were kidding    Data Access   In your application code  e g  PHP   in whatever DB access method you use  you ll need to set the connection charset to utf8mb4   This way  MySQL does no conversion from its native UTF-8 when it hands data off to your application and vice versa   Some drivers provide their own mechanism for configuring the connection character set  which both updates its own internal state and informs MySQL of the encoding to be used on the connection   this is usually the preferred approach    In PHP   If you re using the PDO abstraction layer with PHP   5 3 6  you can specify charset in the DSN    dbh   new PDO  mysql charset utf8mb4      If you re using mysqli  you can call set charset       mysqli- gt set charset  utf8mb4             object oriented style   mysqli set charset  link   utf8mb4        procedural style   If you re stuck with plain mysql but happen to be running PHP   5 2 3  you can call mysql set charset     If the driver does not provide its own mechanism for setting the connection character set  you may have to issue a query to tell MySQL how your application expects data on the connection to be encoded  SET NAMES  utf8mb4    The same consideration regarding utf8mb4 utf8 applies as above    Output   If your application transmits text to other systems  they will also need to be informed of the character encoding   With web applications  the browser must be informed of the encoding in which data is sent  through HTTP response headers or HTML metadata    In PHP  you can use the default charset php ini option  or manually issue the Content-Type MIME header yourself  which is just more work but has the same effect   When encoding the output using json encode    add JSON UNESCAPED UNICODE as a second parameter    Input   Unfortunately  you should verify every received string as being valid UTF-8 before you try to store it or use it anywhere   PHP s mb check encoding   does the trick  but you have to use it religiously   There s really no way around this  as malicious clients can submit data in whatever encoding they want  and I haven t found a trick to get PHP to do this for you reliably   From my reading of the current HTML spec  the following sub-bullets are not necessary or even valid anymore for modern HTML   My understanding is that browsers will work with and submit data in the character set specified for the document   However  if you re targeting older versions of HTML  XHTML  HTML4  etc    these points may still be useful   For HTML before HTML5 only  you want all data sent to you by browsers to be in UTF-8   Unfortunately  if you go by the only way to reliably do this is add the accept-charset attribute to all your  lt form gt  tags   lt form     accept-charset  quot UTF-8 quot  gt   For HTML before HTML5 only  note that the W3C HTML spec says that clients  quot should quot  default to sending forms back to the server in whatever charset the server served  but this is apparently only a recommendation  hence the need for being explicit on every single  lt form gt  tag     Other Code Considerations   Obviously enough  all files you ll be serving  PHP  HTML  JavaScript  etc   should be encoded in valid UTF-8   You need to make sure that every time you process a UTF-8 string  you do so safely   This is  unfortunately  the hard part   You ll probably want to make extensive use of PHP s mbstring extension   PHP s built-in string operations are not by default UTF-8 safe   There are some things you can safely do with normal PHP string operations  like concatenation   but for most things you should use the equivalent mbstring function   To know what you re doing  read  not mess it up   you really need to know UTF-8 and how it works on the lowest possible level   Check out any of the links from utf8 com for some good resources to learn everything you need to know

User · Answer

In my case  I was using mb split  which uses regex   Therefore I also had to manually make sure the regex encoding was utf-8 by doing mb regex encoding  UTF-8     As a side note  I also discovered by running mb internal encoding   that the internal encoding wasn t utf-8  and I changed that by running mb internal encoding  UTF-8

User · Answer

I found an issue with someone using PDO and the answer was to use this for the PDO connection string    pdo   new PDO       mysql host mysql example com dbname example db        username        password       array PDO  MYSQL ATTR INIT COMMAND   gt   SET NAMES utf8       The site I took this from is down  but I was able to get it using the Google cache  luckily

User · Answer

If you want MySQL server to decide character set  and not PHP as a client  old behaviour  preferred  in my opinion   try adding skip-character-set-client-handshake to your my cnf  under  mysqld   and restart mysql   This may cause troubles in case you re using anything other than UTF8

User · Answer

Unicode support in PHP is still a huge mess  While it s capable of converting an ISO8859 string  which it uses internally  to utf8  it lacks the capability to work with unicode strings natively  which means all the string processing functions will mangle and corrupt your strings  So you have to either use a separate library for proper utf8 support  or rewrite all the string handling functions yourself   The easy part is just specifying the charset in HTTP headers and in the database and such  but none of that matters if your PHP code doesn t output valid UTF8  That s the hard part  and PHP gives you virtually no help there   I think PHP6 is supposed to fix the worst of this  but that s still a while away

User · Answer

First of all if you are in  lt  5 3PHP then no  You ve got a ton of problems to tackle    I am surprised that none has mentioned the intl library  the one that has good support for unicode  graphemes  string operations   localisation and many more  see below    I will quote some information about unicode support in PHP by Elizabeth Smith s slides at PHPBenelux 14  INTL  Good    Wrapper around ICU library Standardised locales  set locale per script Number formatting Currency formatting Message formatting  replaces gettext  Calendars  dates  timezone and time Transliterator Spoofchecker Resource bundles Convertors IDN support Graphemes Collation Iterators   Bad    Does not support zend multibite Does not support HTTP input output conversion Does not support function overloading   mb string   Enables zend multibyte support Supports transparent HTTP in out encoding Provides some wrappers for funtionallity such as strtoupper   ICONV   Primary for charset conversion Output buffer handler mime encoding functionality conversion some string helpers  len  substr  strpos  strrpos  Stream Filter stream filter append  fp   convert iconv ISO-2022-JP EUC-JP     DATABASES   mysql  Charset and collation on tables and on connection  not the collation   Also don t use mysql - msqli or PDO postgresql  pg set client encoding sqlite 3   Make sure it was compiled with unicode and intl support   Some other Gotchas   You cannot use unicode filenames with PHP and windows unless you use a 3rd part extension   Send everything in ASCII if you are using exec  proc open and other command line calls Plain text is not plain text  files have encodings You can convert files on the fly with the iconv filter   I ll update this answer in case things change features added and so on

User · Answer

I d like to add one thing to chazomaticus  excellent answer   Don t forget the META tag either  like this  or the HTML4 or XHTML version of it     lt meta charset  utf-8  gt    That seems trivial  but IE7 has given me problems with that before   I was doing everything right  the database  database connection and Content-Type HTTP header were all set to UTF-8  and it worked fine in all other browsers  but Internet Explorer still insisted on using the  Western European  encoding   It turned out the page was missing the META tag  Adding that solved the problem   Edit   The W3C actually has a rather large section dedicated to I18N  They have a number of articles related to this issue  ndash  describing the HTTP   X HTML and CSS side of things    FAQ  Changing  X HTML page encoding to UTF-8 Declaring character encodings in HTML Tutorial  Character sets  amp  encodings in XHTML  HTML and CSS Setting the HTTP charset parameter   They recommend using both the HTTP header and HTML meta tag  or XML declaration in case of XHTML served as XML

User · Answer

in connection php  mysqli set charset  con    utf8      and in sql collation utf 8

User · Answer

Just a note   You are facing the problem of your non-latin characters is showing as             you asked a question  and it got closed with a reference to this canonical question  you tried everything and no matter what you do you still get            from MySQL   That is mostly because you are testing on your old data which has been inserted to the database using the wrong charset and got converted and stored to actually the question mark characters    Which means you lost your original text forever and no matter what you try you will get           re applying what you have learned from the answers of this question on a fresh data could solve your problem

User · Answer

Unicode support in PHP is still a huge mess  While it s capable of converting an ISO8859 string  which it uses internally  to utf8  it lacks the capability to work with unicode strings natively  which means all the string processing functions will mangle and corrupt your strings  So you have to either use a separate library for proper utf8 support  or rewrite all the string handling functions yourself   The easy part is just specifying the charset in HTTP headers and in the database and such  but none of that matters if your PHP code doesn t output valid UTF8  That s the hard part  and PHP gives you virtually no help there   I think PHP6 is supposed to fix the worst of this  but that s still a while away

User · Answer

if you want a mysql solution   I had similar issues with 2 of my projects  after a server migration  After searching and trying a lot of solutions i came across with this one  nothing before this one worked    mysqli set charset  con  utf8      After adding  this line to my config file everything works fine   I found this solution https   www w3schools com PHP func mysqli set charset asp when i was looking to solve a insert from html query  good luck

User · Answer

I have just went through the same issue and found a good solution at PHP manuals   I changed all my file encoding to UTF8 then the default encoding on my connection  This solved all the problems   if    mysqli- gt set charset  utf8          printf  Error loading character set utf8   s n    mysqli- gt error     else      printf  Current character set   s n    mysqli- gt character set name         View Source

User · Answer

In PHP  you ll need to either use the multibyte functions  or turn on mbstring func overload  That way things like strlen will work if you have characters that take more than one byte   You ll also need to identify the character set of your responses  You can either use AddDefaultCharset  as above  or write PHP code that returns the header   Or you can add a META tag to your HTML documents

User · Answer

Unicode support in PHP is still a huge mess  While it s capable of converting an ISO8859 string  which it uses internally  to utf8  it lacks the capability to work with unicode strings natively  which means all the string processing functions will mangle and corrupt your strings  So you have to either use a separate library for proper utf8 support  or rewrite all the string handling functions yourself   The easy part is just specifying the charset in HTTP headers and in the database and such  but none of that matters if your PHP code doesn t output valid UTF8  That s the hard part  and PHP gives you virtually no help there   I think PHP6 is supposed to fix the worst of this  but that s still a while away

User · Answer

I have just went through the same issue and found a good solution at PHP manuals   I changed all my file encoding to UTF8 then the default encoding on my connection  This solved all the problems   if    mysqli- gt set charset  utf8          printf  Error loading character set utf8   s n    mysqli- gt error     else      printf  Current character set   s n    mysqli- gt character set name         View Source

User · Answer

I d like to add one thing to chazomaticus  excellent answer   Don t forget the META tag either  like this  or the HTML4 or XHTML version of it     lt meta charset  utf-8  gt    That seems trivial  but IE7 has given me problems with that before   I was doing everything right  the database  database connection and Content-Type HTTP header were all set to UTF-8  and it worked fine in all other browsers  but Internet Explorer still insisted on using the  Western European  encoding   It turned out the page was missing the META tag  Adding that solved the problem   Edit   The W3C actually has a rather large section dedicated to I18N  They have a number of articles related to this issue  ndash  describing the HTTP   X HTML and CSS side of things    FAQ  Changing  X HTML page encoding to UTF-8 Declaring character encodings in HTML Tutorial  Character sets  amp  encodings in XHTML  HTML and CSS Setting the HTTP charset parameter   They recommend using both the HTTP header and HTML meta tag  or XML declaration in case of XHTML served as XML

User · Answer

The top answer is excellent  Here is what I had to on a regular debian php mysql setup      storage    debian  apparently already utf-8     retrieval    the mysql database was stored in utf-8      but apparently php was requesting iso  this worked         notice  utf8   without dash  this is a mysql encoding    mysql set charset  utf8        delivery    php ini did not have a default charset       it was commented out  shared host  and    no http encoding was specified in the apache headers     this made apache send out a utf-8 header     and perhaps made php actually send out utf-8        notice  utf-8   with dash  this is a php encoding    ini set  default charset   utf-8        submission    this worked in all major browsers once apache    was sending out the utf-8 header  i didnt add    the accept-charset attribute      processing    changed a few commands in php  like substr     to mb substr   that was all

User · Answer

I found an issue with someone using PDO and the answer was to use this for the PDO connection string    pdo   new PDO       mysql host mysql example com dbname example db        username        password       array PDO  MYSQL ATTR INIT COMMAND   gt   SET NAMES utf8       The site I took this from is down  but I was able to get it using the Google cache  luckily

User · Answer

The only thing I would add to these amazing answers is to emphasize on saving your files in utf8 encoding  i have noticed that browsers accept this property over setting utf8 as your code encoding  Any decent text editor will show you this  for example Notepad   has a menu option for file enconding  it shows you the current encoding and enables you to change it  For all my php files I use utf8 without BOM    Sometime ago i had someone ask me to add utf8 support for a php mysql application designed by someone else  i noticed that all files were encoded in ANSI  so I had to use ICONV to convert all files  change the database tables to use the utf8 charset and utf8 general ci collate  add  SET NAMES utf8  to the database abstraction layer after the connection  if using 5 3 6 or earlier otherwise you have to use charset utf8 in the connection string  and change string functions to use the php multibyte string functions equivalent

User · Answer

In PHP  you ll need to either use the multibyte functions  or turn on mbstring func overload  That way things like strlen will work if you have characters that take more than one byte   You ll also need to identify the character set of your responses  You can either use AddDefaultCharset  as above  or write PHP code that returns the header   Or you can add a META tag to your HTML documents

[php] UTF-8 all the way through

Examples related to php

Examples related to mysql

Examples related to linux

Examples related to apache

Examples related to utf-8