[php] Why does the PHP json_encode function convert UTF-8 strings to hexadecimal entities?

I have a PHP script that deals with a wide variety of languages. Unfortunately, whenever I try to use json_encode, any Unicode output is converted to hexadecimal entities. Is this the expected behavior? Is there any way to convert the output to UTF-8 characters?

Here's an example of what I'm seeing:

INPUT

echo $text;

OUTPUT

???? ????? ??????.

INPUT

json_encode($text);

OUTPUT

"\u0411\u0430\u0437\u0430 \u0434\u0430\u043d\u043d\u0438 \u0433\u0440\u0435\u0448\u043a\u0430."

Is this the expected behavior?

the json_encode() only works with UTF-8 encoded data.

maybe you can get an answer to convert it here: cyrillic-characters-in-phps-json-encode

One solution is to first encode data and then decode it in the same file:

$string =json_encode($input, JSON_UNESCAPED_UNICODE) ; 
echo $decoded = html_entity_decode( $string );

Here is my combined solution for various PHP versions.

In my company we are working with different servers with various PHP versions, so I had to find solution working for all.

$phpVersion = substr(phpversion(), 0, 3)*1;

if($phpVersion >= 5.4) {
  $encodedValue = json_encode($value, JSON_UNESCAPED_UNICODE);
} else {
  $encodedValue = preg_replace('/\\\\u([a-f0-9]{4})/e', "iconv('UCS-4LE','UTF-8',pack('V', hexdec('U$1')))", json_encode($value));
}

Credits should go to Marco Gasi & abu. The solution for PHP >= 5.4 is provided in the json_encode docs.

You like to set charset and unescaped unicode

 header('Content-Type: application/json;charset=utf-8');  
 json_encode($data,JSON_UNESCAPED_UNICODE|JSON_PRETTY_PRINT);

JSON_UNESCAPED_UNICODE is available on PHP Version 5.4 or later.
The following code is for Version 5.3.

UPDATED

• html_entity_decode is a bit more efficient than pack + mb_convert_encoding.
• (*SKIP)(*FAIL) skips backslashes itself and specified characters by JSON_HEX_* flags.

function raw_json_encode($input, $flags = 0) {
    $fails = implode('|', array_filter(array(
        '\\\\',
        $flags & JSON_HEX_TAG ? 'u003[CE]' : '',
        $flags & JSON_HEX_AMP ? 'u0026' : '',
        $flags & JSON_HEX_APOS ? 'u0027' : '',
        $flags & JSON_HEX_QUOT ? 'u0022' : '',
    )));
    $pattern = "/\\\\(?:(?:$fails)(*SKIP)(*FAIL)|u([0-9a-fA-F]{4}))/";
    $callback = function ($m) {
        return html_entity_decode("&#x$m[1];", ENT_QUOTES, 'UTF-8');
    };
    return preg_replace_callback($pattern, $callback, json_encode($input, $flags));
}

The raw_json_encode() function above did not solve me the problem (for some reason, the callback function raised an error on my PHP 5.2.5 server).

But this other solution did actually work.

https://www.experts-exchange.com/questions/28628085/json-encode-fails-with-special-characters.html

Credits should go to Marco Gasi. I just call his function instead of calling json_encode():

function jsonRemoveUnicodeSequences( $json_struct )
{ 
    return preg_replace( "/\\\\u([a-f0-9]{4})/e", "iconv('UCS-4LE','UTF-8',pack('V', hexdec('U$1')))", json_encode( $json_struct ) );
}

json_encode($text, JSON_UNESCAPED_UNICODE|JSON_UNESCAPED_SLASHES);