UConverter can be used since PHP 5.5. UConverter is better the choice if you use intl extension and don't use mbstring.
function replace_invalid_byte_sequence($str)
{
return UConverter::transcode($str, 'UTF-8', 'UTF-8');
}
function replace_invalid_byte_sequence2($str)
{
return (new UConverter('UTF-8', 'UTF-8'))->convert($str);
}
htmlspecialchars can be used to remove invalid byte sequence since PHP 5.4. Htmlspecialchars is better than preg_match for handling large size of byte and the accuracy. A lot of the wrong implementation by using regular expression can be seen.
function replace_invalid_byte_sequence3($str)
{
return htmlspecialchars_decode(htmlspecialchars($str, ENT_SUBSTITUTE, 'UTF-8'));
}
Welcome to 2019 and the /u
modifier in regex which will handle UTF-8 multibyte chars for you
If you only use mb_convert_encoding($value, 'UTF-8', 'UTF-8')
you will still end up with non-printable chars in your string
This method will:
mb_convert_encoding
\r
, \x00
(NULL-byte) and other control chars with preg_replace
function utf8_filter(string $value): string{
return preg_replace('/[^[:print:]\n]/u', '', mb_convert_encoding($value, 'UTF-8', 'UTF-8'));
}
[:print:]
match all printable chars and \n
newlines and strip everything else
You can see the ASCII table below.. The printable chars range from 32 to 127, but newline \n
is a part of the control chars which range from 0 to 31 so we have to add newline to the regex /[^[:print:]\n]/u
You can try to send strings through the regex with chars outside the printable range like \x7F
(DEL), \x1B
(Esc) etc. and see how they are stripped
function utf8_filter(string $value): string{
return preg_replace('/[^[:print:]\n]/u', '', mb_convert_encoding($value, 'UTF-8', 'UTF-8'));
}
$arr = [
'Danish chars' => 'Hello from Denmark with æøå',
'Non-printable chars' => "\x7FHello with invalid chars\r \x00"
];
foreach($arr as $k => $v){
echo "$k:\n---------\n";
$len = strlen($v);
echo "$v\n(".$len.")\n";
$strip = utf8_decode(utf8_filter(utf8_encode($v)));
$strip_len = strlen($strip);
echo $strip."\n(".$strip_len.")\n\n";
echo "Chars removed: ".($len - $strip_len)."\n\n\n";
}
How about iconv:
http://php.net/manual/en/function.iconv.php
Haven't used it inside PHP itself but its always performed well for me on the command line. You can get it to substitute invalid characters.
You can use mbstring:
$text = mb_convert_encoding($text, 'UTF-8', 'UTF-8');
...will remove invalid characters.
See: Replacing invalid UTF-8 characters by question marks, mbstring.substitute_character seems ignored
$string = preg_replace('~&([a-z]{1,2})(acute|cedil|circ|grave|lig|orn|ring|slash|th|tilde|uml);~i', '$1', htmlentities($string, ENT_COMPAT, 'UTF-8'));
static $preg = <<<'END'
%(
[\x09\x0A\x0D\x20-\x7E]
| [\xC2-\xDF][\x80-\xBF]
| \xE0[\xA0-\xBF][\x80-\xBF]
| [\xE1-\xEC\xEE\xEF][\x80-\xBF]{2}
| \xED[\x80-\x9F][\x80-\xBF]
| \xF0[\x90-\xBF][\x80-\xBF]{2}
| [\xF1-\xF3][\x80-\xBF]{3}
| \xF4[\x80-\x8F][\x80-\xBF]{2}
)%xs
END;
if (preg_match_all($preg, $string, $match)) {
$string = implode('', $match[0]);
} else {
$string = '';
}
it work on our service
Slightly different to the question, but what I am doing is to use HtmlEncode(string),
pseudo code here
var encoded = HtmlEncode(string);
encoded = Regex.Replace(encoded, "&#\d+?;", "");
var result = HtmlDecode(encoded);
input and output
"Headlight\x007E Bracket, { Cafe Racer<> Style, Stainless Steel ????"
"Headlight~ Bracket, { Cafe Racer<> Style, Stainless Steel ????"
I know it's not perfect, but does the job for me.
If you apply utf8_encode()
to an already UTF8 string it will return a garbled UTF8 output.
I made a function that addresses all this issues. It´s called Encoding::toUTF8()
.
You dont need to know what the encoding of your strings is. It can be Latin1 (ISO8859-1), Windows-1252 or UTF8, or the string can have a mix of them. Encoding::toUTF8()
will convert everything to UTF8.
I did it because a service was giving me a feed of data all messed up, mixing those encodings in the same string.
Usage:
require_once('Encoding.php');
use \ForceUTF8\Encoding; // It's namespaced now.
$utf8_string = Encoding::toUTF8($mixed_string);
$latin1_string = Encoding::toLatin1($mixed_string);
I've included another function, Encoding::fixUTF8(), which will fix every UTF8 string that looks garbled product of having been encoded into UTF8 multiple times.
Usage:
require_once('Encoding.php');
use \ForceUTF8\Encoding; // It's namespaced now.
$utf8_string = Encoding::fixUTF8($garbled_utf8_string);
Examples:
echo Encoding::fixUTF8("Fédération Camerounaise de Football");
echo Encoding::fixUTF8("Fédération Camerounaise de Football");
echo Encoding::fixUTF8("FÃÂédÃÂération Camerounaise de Football");
echo Encoding::fixUTF8("Fédération Camerounaise de Football");
will output:
Fédération Camerounaise de Football
Fédération Camerounaise de Football
Fédération Camerounaise de Football
Fédération Camerounaise de Football
Download:
substr() can break your multi-byte characters!
In my case, I was using substr($string, 0, 255)
to ensure a user supplied value would fit in the database. On occasion it would split a multi-byte character in half and caused database errors with "Incorrect string value".
You could use mb_substr($string,0,255)
, and it might be ok for MySQL 5, but MySQL 4 counts bytes instead of characters, so it would still be too long depending on the number of multi-byte characters.
To prevent these issues I implemented the following steps:
mb_substring
in case it was still too longI have made a function that deletes invalid UTF-8 characters from a string. I'm using it to clear description of 27000 products before it generates the XML export file.
public function stripInvalidXml($value) {
$ret = "";
$current;
if (empty($value)) {
return $ret;
}
$length = strlen($value);
for ($i=0; $i < $length; $i++) {
$current = ord($value{$i});
if (($current == 0x9) || ($current == 0xA) || ($current == 0xD) || (($current >= 0x20) && ($current <= 0xD7FF)) || (($current >= 0xE000) && ($current <= 0xFFFD)) || (($current >= 0x10000) && ($current <= 0x10FFFF))) {
$ret .= chr($current);
}
else {
$ret .= "";
}
}
return $ret;
}
To remove all Unicode characters outside of the Unicode basic language plane:
$str = preg_replace("/[^\\x00-\\xFFFF]/", "", $str);
try this:
$string = iconv("UTF-8","UTF-8//IGNORE",$string);
According to the iconv manual, the function will take the first parameter as the input charset, second parameter as the output charset, and the third as the actual input string.
If you set both the input and output charset to UTF-8, and append the //IGNORE
flag to the output charset, the function will drop(strip) all characters in the input string that can't be represented by the output charset. Thus, filtering the input string in effect.
From recent patch to Drupal's Feeds JSON parser module:
//remove everything except valid letters (from any language)
$raw = preg_replace('/(?:\\\\u[\pL\p{Zs}])+/', '', $raw);
If you're concerned yes it retains spaces as valid characters.
Did what I needed. It removes widespread nowadays emoji-characters that don't fit into MySQL's 'utf8' character set and that gave me errors like "SQLSTATE[HY000]: General error: 1366 Incorrect string value".
For details see https://www.drupal.org/node/1824506#comment-6881382
Maybe not the most precise solution, but it gets the job done with a single line of code:
echo str_replace("?","",(utf8_decode($str)));
utf8_decode
will convert the characters to a question mark;
str_replace
will strip out the question marks.
This function removes all NON ASCII characters, it's useful but not solving the question:
This is my function that always works, regardless of encoding:
function remove_bs($Str) {
$StrArr = str_split($Str); $NewStr = '';
foreach ($StrArr as $Char) {
$CharNo = ord($Char);
if ($CharNo == 163) { $NewStr .= $Char; continue; } // keep £
if ($CharNo > 31 && $CharNo < 127) {
$NewStr .= $Char;
}
}
return $NewStr;
}
How it works:
echo remove_bs('Hello õhowå åare youÆ?'); // Hello how are you?
So the rules are that the first UTF-8 octlet has the high bit set as a marker, and then 1 to 4 bits to indicate how many additional octlets; then each of the additional octlets must have the high two bits set to 10.
The pseudo-python would be:
newstring = ''
cont = 0
for each ch in string:
if cont:
if (ch >> 6) != 2: # high 2 bits are 10
# do whatever, e.g. skip it, or skip whole point, or?
else:
# acceptable continuation of multi-octlet char
newstring += ch
cont -= 1
else:
if (ch >> 7): # high bit set?
c = (ch << 1) # strip the high bit marker
while (c & 1): # while the high bit indicates another octlet
c <<= 1
cont += 1
if cont > 4:
# more than 4 octels not allowed; cope with error
if !cont:
# illegal, do something sensible
newstring += ch # or whatever
if cont:
# last utf-8 was not terminated, cope
This same logic should be translatable to php. However, its not clear what kind of stripping is to be done once you get a malformed character.
The text may contain non-utf8 character. Try to do first:
$nonutf8 = mb_convert_encoding($nonutf8 , 'UTF-8', 'UTF-8');
You can read more about it here: http://php.net/manual/en/function.mb-convert-encoding.phpnews
$text = iconv("UTF-8", "UTF-8//IGNORE", $text);
This is what I am using. Seems to work pretty well. Taken from http://planetozh.com/blog/2005/01/remove-invalid-characters-in-utf-8/
Source: Stackoverflow.com