PHP output showing little black diamonds with a question mark

Question

I m writing a php program that pulls from a database source  Some of the varchars have quotes that are displaying as black diamonds with a question mark in them     REPLACEMENT CHARACTER  I assume from Microsoft Word text    How can I use php to strip these characters out

User · Answer

Just add these lines before headers   Accurate format of  doc docx files will be retrieved    if ini get  zlib output compression        ini set  zlib output compression    Off     ob clean

User · Answer

For global purposes  Instead of converting  codifying  decodifying each text I prefer to let them as they are and instead change the server php settings  So   Let the diamonds  From the browser  on the view menu select  quot text encoding quot  and find the one which let s you see your text correctly   Edit your php ini and add  default charset    quot ISO-8859-1 quot    or instead of ISO-8859 the one which fits your text encoding

User · Answer

You can also change the caracter set in your browser  Just for debug reasons

User · Answer

Based on your description of the problem  the data in your database is almost certainly encoded as Windows-1252  and your page is almost certainly being served as ISO-8859-1  These two character sets are equivalent except that Windows-1252 has 16 extra characters which are not present in ISO-8859-1  including left and right curly quotes   Assuming my analysis is correct  the simplest solution is to serve your page as Windows-1252  This will work because all characters that are in ISO-8859-1 are also in Windows-1252  In PHP you can change the encoding as follows   header  Content-Type  text html  charset Windows-1252      However  you really should check what character encoding you are using in your HTML files and the contents of your database  and take care to be consistent  or convert properly where this is not possible

User · Answer

This will help you  Put this inside  lt head gt  tag   lt meta charset  iso-8859-1  gt

User · Answer

If you see that character    U FFFD  REPLACEMENT CHARACTER   it usually means that the text itself is encoded in some form of single byte encoding but interpreted in one of the unicode encodings  UTF8 or UTF16    If it were the other way around it would  usually  look something like this         Probably the original encoding is ISO-8859-1  also known as Latin-1  You can check this without having to change your script  Browsers give you the option to re-interpret a page in a different encoding -- in Firefox use  View  -   Character Encoding     To make the browser use the correct encoding  add an HTTP header like this   header  Content-Type  text html  charset ISO-8859-1      or put the encoding in a meta tag    lt meta http-equiv  Content-Type  content  text html  charset iso-8859-1  gt    Alternatively you could try to read from the database in another encoding  UTF-8  preferably  or convert the text with iconv

User · Answer

Try This Please  mb substr  description  0  490   UTF-8

User · Answer

You can also change the caracter set in your browser  Just for debug reasons

User · Answer

That can be caused by unicode or other charset mismatch  Try changing charset in your browser  in of the settings the text will look OK  Then it s question of how to convert your database contents to charset you use for displaying   Which can actually be just adding utf-8 charset statement to your output

User · Answer

Add this function to your variables  utf8 encode  your variable

User · Answer

If you see that character    U FFFD  REPLACEMENT CHARACTER   it usually means that the text itself is encoded in some form of single byte encoding but interpreted in one of the unicode encodings  UTF8 or UTF16    If it were the other way around it would  usually  look something like this         Probably the original encoding is ISO-8859-1  also known as Latin-1  You can check this without having to change your script  Browsers give you the option to re-interpret a page in a different encoding -- in Firefox use  View  -   Character Encoding     To make the browser use the correct encoding  add an HTTP header like this   header  Content-Type  text html  charset ISO-8859-1      or put the encoding in a meta tag    lt meta http-equiv  Content-Type  content  text html  charset iso-8859-1  gt    Alternatively you could try to read from the database in another encoding  UTF-8  preferably  or convert the text with iconv

User · Answer

This happened to work in my case    text   utf8 decode  text    I turns the black diamond character into a question mark so you can     text   str replace          utf8 decode  text

User · Answer

I chose to strip these characters out of the string by doing this -   ini set  mbstring substitute character    none      text  mb convert encoding  text   UTF-8    UTF-8

User · Answer

This is a charset issue  As such  it can have gone wrong on many different levels  but most likely  the strings in your database are utf-8 encoded  and you are presenting them as iso-8859-1  Or the other way around  The proper way to fix this problem  is to get your character-sets straight  The simplest strategy  since you re using PHP  is to use iso-8859-1 throughout your application  To do this  you must ensure that   All PHP source-files are saved as iso-8859-1  Not to be confused with cp-1252   Your web-server is configured to serve files with charset iso-8859-1 Alternatively  you can override the webservers settings from within the PHP-document  using header  In addition  you may insert a meta-tag in you HTML  that specifies the same thing  but this isn t strictly needed  You may also specify the accept-charset attribute on your  lt form gt  elements  Database tables are defined with encoding as latin1 The database connection between PHP to and database is set to latin1  If you already have data in your database  you should be aware that they are probably messed up already  If you are not already in production phase  just wipe it all and start over  Otherwise you ll have to do some data cleanup  A note on meta-tags  since everybody misunderstands what they are  When a web-server serves a file  A HTML-document   it sends some information  that isn t presented directly in the browser  This is known as HTTP-headers  One such header  is the Content-Type header  which specifies the mimetype of the file  Eg  text html  as well as the encoding  aka charset   While most webservers will send a Content-Type header with charset info  it s optional  If it isn t present  the browser will instead interpret any meta-tags with http-equiv  quot Content-Type quot   It s important to realise that the meta-tag is only interpreted if the webserver doesn t send the header  In practice this means that it s only used if the page is saved to disk and then opened from there  This page has a very good explanation of these things

User · Answer

When you extract data from anywhere you should use functions with the prefix md FUNC NAME   Had the same problem it helped me out   Or you can find the code of this symbol and use regexp to delete these symbols

User · Answer

You can also change the caracter set in your browser  Just for debug reasons

User · Answer

Based on your description of the problem  the data in your database is almost certainly encoded as Windows-1252  and your page is almost certainly being served as ISO-8859-1  These two character sets are equivalent except that Windows-1252 has 16 extra characters which are not present in ISO-8859-1  including left and right curly quotes   Assuming my analysis is correct  the simplest solution is to serve your page as Windows-1252  This will work because all characters that are in ISO-8859-1 are also in Windows-1252  In PHP you can change the encoding as follows   header  Content-Type  text html  charset Windows-1252      However  you really should check what character encoding you are using in your HTML files and the contents of your database  and take care to be consistent  or convert properly where this is not possible

User · Answer

what I ended up doing in the end after I fixed my tables was to back it up and change back the settings to utf-8 then I altered my dump file so that DEFAULT CHARACTER SET utf8   COLLATE utf8 general ci  are my character set entries   now I don t have characterset issues anymore because the database and browser are utf8   I figured out what caused it  It was the web page browser effects on the DB  On the terminals that are linux  ubuntu firefox  it was encoding the database in latin1 which is what the tabes are set  But on the windows 10 edge terminals  the entries were force coded into utf8  Also I noticed the windows 10 has issues staying with latin1 so I decided to bend with the wind and convert all to utf8   I figured it was a windows 10 issue because we started using win 10 terminals  so yet again microsoft bugs causes issues  I still don t know why the encoding changes on the forms because the browser in windows 10 shows the latin1 characterset but when it goes in its utf8 encoded and I get the data anomaly  but in linux firefox it doesn t do that

User · Answer

I ran  the  detect encoding  code after my collation change in phpmyadmin and now it comes up as Latin 1   but here is something I came across looking a different data anomaly in my application and how I fixed it   I just imported a table that has mixed encoding  with diamond question marks in some lines  and all were in the same column   so here is my fix code  I used utf8 decode process that takes the undefined placeholder and assigns a plain question mark in the place of the  diamond question mark   then I used str replace to replace the question mark with a space between quotes  here is the   code              include  dbconnectfile php           the variable  db comes from my db connect file        inx is my auto increment column        broke column is the column I need to fix         qwy    select inx broke column from Table           res    db- gt query  qwy           while   data    res- gt fetch row            for   m 0   m lt  res- gt field count   m                 if   m  0                id 0              id  data  m          echo  id              else if   m  1                  fix 0                fix  data  m                   fix   utf8 decode  fix                 fixx  str replace          fix            echo  fixx               I echoed the data to the screen because I like to see something as I execute it                                          insert   UPDATE Table SET broke column     fixx     where inx     id                 insresult   db- gt query  insert         echo  lt br gt                         gt

User · Answer

When you extract data from anywhere you should use functions with the prefix md FUNC NAME   Had the same problem it helped me out   Or you can find the code of this symbol and use regexp to delete these symbols

User · Answer

That can be caused by unicode or other charset mismatch  Try changing charset in your browser  in of the settings the text will look OK  Then it s question of how to convert your database contents to charset you use for displaying   Which can actually be just adding utf-8 charset statement to your output

User · Answer

Add this function to your variables  utf8 encode  your variable

User · Answer

This happened to work in my case    text   utf8 decode  text    I turns the black diamond character into a question mark so you can     text   str replace          utf8 decode  text

User · Answer

I chose to strip these characters out of the string by doing this -   ini set  mbstring substitute character    none      text  mb convert encoding  text   UTF-8    UTF-8

User · Answer

Using the same charset  as suggested here  in both the database and the HTML has not worked for me    So remembering that the code is generated as HTML  I chose to use the  amp quot  HTML code  or the  amp  34   ISO Latin-1 code  in my database text where quotes were used   This solved the problem while providing me a quotation mark   It is odd to note that prior to this solution  only some of the quotation marks and apostrophes did not display correctly while others did  however  the special code did work in all instances

User · Answer

To make sure your MYSQL connection is set to UTF-8  or latin1  depending on what you re using   you can do this to    con   mysql connect  localhost   username   password        mysql set charset  utf8   con     or use this to check what charset you are using     con   mysql connect  localhost   username   password        charset   mysql client encoding  con   echo  The current character set is   charset n      More info here  http   php net manual en function mysql-set-charset php

User · Answer

As mentioned in earlier answers  it is happening because your text has been written to the database in iso-8859-1 encoding  or any other format   So you just need to convert the data to utf8 before outputting it     text      string from database      text   utf8 encode  text   echo  text

User · Answer

This is a charset issue  As such  it can have gone wrong on many different levels  but most likely  the strings in your database are utf-8 encoded  and you are presenting them as iso-8859-1  Or the other way around  The proper way to fix this problem  is to get your character-sets straight  The simplest strategy  since you re using PHP  is to use iso-8859-1 throughout your application  To do this  you must ensure that   All PHP source-files are saved as iso-8859-1  Not to be confused with cp-1252   Your web-server is configured to serve files with charset iso-8859-1 Alternatively  you can override the webservers settings from within the PHP-document  using header  In addition  you may insert a meta-tag in you HTML  that specifies the same thing  but this isn t strictly needed  You may also specify the accept-charset attribute on your  lt form gt  elements  Database tables are defined with encoding as latin1 The database connection between PHP to and database is set to latin1  If you already have data in your database  you should be aware that they are probably messed up already  If you are not already in production phase  just wipe it all and start over  Otherwise you ll have to do some data cleanup  A note on meta-tags  since everybody misunderstands what they are  When a web-server serves a file  A HTML-document   it sends some information  that isn t presented directly in the browser  This is known as HTTP-headers  One such header  is the Content-Type header  which specifies the mimetype of the file  Eg  text html  as well as the encoding  aka charset   While most webservers will send a Content-Type header with charset info  it s optional  If it isn t present  the browser will instead interpret any meta-tags with http-equiv  quot Content-Type quot   It s important to realise that the meta-tag is only interpreted if the webserver doesn t send the header  In practice this means that it s only used if the page is saved to disk and then opened from there  This page has a very good explanation of these things

User · Answer

Try This Please  mb substr  description  0  490   UTF-8

User · Answer

If you see that character    U FFFD  REPLACEMENT CHARACTER   it usually means that the text itself is encoded in some form of single byte encoding but interpreted in one of the unicode encodings  UTF8 or UTF16    If it were the other way around it would  usually  look something like this         Probably the original encoding is ISO-8859-1  also known as Latin-1  You can check this without having to change your script  Browsers give you the option to re-interpret a page in a different encoding -- in Firefox use  View  -   Character Encoding     To make the browser use the correct encoding  add an HTTP header like this   header  Content-Type  text html  charset ISO-8859-1      or put the encoding in a meta tag    lt meta http-equiv  Content-Type  content  text html  charset iso-8859-1  gt    Alternatively you could try to read from the database in another encoding  UTF-8  preferably  or convert the text with iconv

User · Answer

You can also change the caracter set in your browser  Just for debug reasons

User · Answer

I also faced this   issue  Meanwhile I ran into three cases where it happened   substr   I was using substr   on a UTF8 string which cut UTF8 characters  thus the cut chars could not be displayed correctly  Use mb substr  utfstring  0  10   utf-8    instead  Credits  htmlspecialchars   Another problem was using htmlspecialchars   on a UTF8 string  The fix is to use  htmlspecialchars  utfstring  ENT QUOTES   UTF-8     preg replace   Lastly I found out that preg replace   can lead to problems with UTF  The code  string   preg replace     A-Za-z0-9                         string   for example transformed the UTF string  quot F     2  -3 quot  into  quot F   2   quot   The fix is to use mb ereg replace   instead    I hope this additional information will help to get rid of such problems

User · Answer

That can be caused by unicode or other charset mismatch  Try changing charset in your browser  in of the settings the text will look OK  Then it s question of how to convert your database contents to charset you use for displaying   Which can actually be just adding utf-8 charset statement to your output

User · Answer

Go to your phpmyadmin and select your database and just increase the length value of that table s field to 500 or 1000 it will solve your problem

User · Answer

Using the same charset  as suggested here  in both the database and the HTML has not worked for me    So remembering that the code is generated as HTML  I chose to use the  amp quot  HTML code  or the  amp  34   ISO Latin-1 code  in my database text where quotes were used   This solved the problem while providing me a quotation mark   It is odd to note that prior to this solution  only some of the quotation marks and apostrophes did not display correctly while others did  however  the special code did work in all instances

User · Answer

To make sure your MYSQL connection is set to UTF-8  or latin1  depending on what you re using   you can do this to    con   mysql connect  localhost   username   password        mysql set charset  utf8   con     or use this to check what charset you are using     con   mysql connect  localhost   username   password        charset   mysql client encoding  con   echo  The current character set is   charset n      More info here  http   php net manual en function mysql-set-charset php

User · Answer

I also faced this   issue  Meanwhile I ran into three cases where it happened   substr   I was using substr   on a UTF8 string which cut UTF8 characters  thus the cut chars could not be displayed correctly  Use mb substr  utfstring  0  10   utf-8    instead  Credits  htmlspecialchars   Another problem was using htmlspecialchars   on a UTF8 string  The fix is to use  htmlspecialchars  utfstring  ENT QUOTES   UTF-8     preg replace   Lastly I found out that preg replace   can lead to problems with UTF  The code  string   preg replace     A-Za-z0-9                         string   for example transformed the UTF string  quot F     2  -3 quot  into  quot F   2   quot   The fix is to use mb ereg replace   instead    I hope this additional information will help to get rid of such problems

User · Answer

Based on your description of the problem  the data in your database is almost certainly encoded as Windows-1252  and your page is almost certainly being served as ISO-8859-1  These two character sets are equivalent except that Windows-1252 has 16 extra characters which are not present in ISO-8859-1  including left and right curly quotes   Assuming my analysis is correct  the simplest solution is to serve your page as Windows-1252  This will work because all characters that are in ISO-8859-1 are also in Windows-1252  In PHP you can change the encoding as follows   header  Content-Type  text html  charset Windows-1252      However  you really should check what character encoding you are using in your HTML files and the contents of your database  and take care to be consistent  or convert properly where this is not possible

User · Answer

Just Paste This Code In Starting to The Top of Page    lt  php header  Content-Type  text html  charset ISO-8859-1      gt

User · Answer

Go to your phpmyadmin and select your database and just increase the length value of that table s field to 500 or 1000 it will solve your problem

User · Answer

That can be caused by unicode or other charset mismatch  Try changing charset in your browser  in of the settings the text will look OK  Then it s question of how to convert your database contents to charset you use for displaying   Which can actually be just adding utf-8 charset statement to your output

User · Answer

what I ended up doing in the end after I fixed my tables was to back it up and change back the settings to utf-8 then I altered my dump file so that DEFAULT CHARACTER SET utf8   COLLATE utf8 general ci  are my character set entries   now I don t have characterset issues anymore because the database and browser are utf8   I figured out what caused it  It was the web page browser effects on the DB  On the terminals that are linux  ubuntu firefox  it was encoding the database in latin1 which is what the tabes are set  But on the windows 10 edge terminals  the entries were force coded into utf8  Also I noticed the windows 10 has issues staying with latin1 so I decided to bend with the wind and convert all to utf8   I figured it was a windows 10 issue because we started using win 10 terminals  so yet again microsoft bugs causes issues  I still don t know why the encoding changes on the forms because the browser in windows 10 shows the latin1 characterset but when it goes in its utf8 encoded and I get the data anomaly  but in linux firefox it doesn t do that

User · Answer

For global purposes  Instead of converting  codifying  decodifying each text I prefer to let them as they are and instead change the server php settings  So   Let the diamonds  From the browser  on the view menu select  quot text encoding quot  and find the one which let s you see your text correctly   Edit your php ini and add  default charset    quot ISO-8859-1 quot    or instead of ISO-8859 the one which fits your text encoding

User · Answer

Just Paste This Code In Starting to The Top of Page    lt  php header  Content-Type  text html  charset ISO-8859-1      gt

User · Answer

Based on your description of the problem  the data in your database is almost certainly encoded as Windows-1252  and your page is almost certainly being served as ISO-8859-1  These two character sets are equivalent except that Windows-1252 has 16 extra characters which are not present in ISO-8859-1  including left and right curly quotes   Assuming my analysis is correct  the simplest solution is to serve your page as Windows-1252  This will work because all characters that are in ISO-8859-1 are also in Windows-1252  In PHP you can change the encoding as follows   header  Content-Type  text html  charset Windows-1252      However  you really should check what character encoding you are using in your HTML files and the contents of your database  and take care to be consistent  or convert properly where this is not possible

User · Answer

This will help you  Put this inside  lt head gt  tag   lt meta charset  iso-8859-1  gt

User · Answer

I ran  the  detect encoding  code after my collation change in phpmyadmin and now it comes up as Latin 1   but here is something I came across looking a different data anomaly in my application and how I fixed it   I just imported a table that has mixed encoding  with diamond question marks in some lines  and all were in the same column   so here is my fix code  I used utf8 decode process that takes the undefined placeholder and assigns a plain question mark in the place of the  diamond question mark   then I used str replace to replace the question mark with a space between quotes  here is the   code              include  dbconnectfile php           the variable  db comes from my db connect file        inx is my auto increment column        broke column is the column I need to fix         qwy    select inx broke column from Table           res    db- gt query  qwy           while   data    res- gt fetch row            for   m 0   m lt  res- gt field count   m                 if   m  0                id 0              id  data  m          echo  id              else if   m  1                  fix 0                fix  data  m                   fix   utf8 decode  fix                 fixx  str replace          fix            echo  fixx               I echoed the data to the screen because I like to see something as I execute it                                          insert   UPDATE Table SET broke column     fixx     where inx     id                 insresult   db- gt query  insert         echo  lt br gt                         gt

User · Answer

Just add these lines before headers   Accurate format of  doc docx files will be retrieved    if ini get  zlib output compression        ini set  zlib output compression    Off     ob clean

User · Answer

As mentioned in earlier answers  it is happening because your text has been written to the database in iso-8859-1 encoding  or any other format   So you just need to convert the data to utf8 before outputting it     text      string from database      text   utf8 encode  text   echo  text

User · Answer

This is a charset issue  As such  it can have gone wrong on many different levels  but most likely  the strings in your database are utf-8 encoded  and you are presenting them as iso-8859-1  Or the other way around  The proper way to fix this problem  is to get your character-sets straight  The simplest strategy  since you re using PHP  is to use iso-8859-1 throughout your application  To do this  you must ensure that   All PHP source-files are saved as iso-8859-1  Not to be confused with cp-1252   Your web-server is configured to serve files with charset iso-8859-1 Alternatively  you can override the webservers settings from within the PHP-document  using header  In addition  you may insert a meta-tag in you HTML  that specifies the same thing  but this isn t strictly needed  You may also specify the accept-charset attribute on your  lt form gt  elements  Database tables are defined with encoding as latin1 The database connection between PHP to and database is set to latin1  If you already have data in your database  you should be aware that they are probably messed up already  If you are not already in production phase  just wipe it all and start over  Otherwise you ll have to do some data cleanup  A note on meta-tags  since everybody misunderstands what they are  When a web-server serves a file  A HTML-document   it sends some information  that isn t presented directly in the browser  This is known as HTTP-headers  One such header  is the Content-Type header  which specifies the mimetype of the file  Eg  text html  as well as the encoding  aka charset   While most webservers will send a Content-Type header with charset info  it s optional  If it isn t present  the browser will instead interpret any meta-tags with http-equiv  quot Content-Type quot   It s important to realise that the meta-tag is only interpreted if the webserver doesn t send the header  In practice this means that it s only used if the page is saved to disk and then opened from there  This page has a very good explanation of these things

User · Answer

If you see that character    U FFFD  REPLACEMENT CHARACTER   it usually means that the text itself is encoded in some form of single byte encoding but interpreted in one of the unicode encodings  UTF8 or UTF16    If it were the other way around it would  usually  look something like this         Probably the original encoding is ISO-8859-1  also known as Latin-1  You can check this without having to change your script  Browsers give you the option to re-interpret a page in a different encoding -- in Firefox use  View  -   Character Encoding     To make the browser use the correct encoding  add an HTTP header like this   header  Content-Type  text html  charset ISO-8859-1      or put the encoding in a meta tag    lt meta http-equiv  Content-Type  content  text html  charset iso-8859-1  gt    Alternatively you could try to read from the database in another encoding  UTF-8  preferably  or convert the text with iconv

User · Answer

This is a charset issue  As such  it can have gone wrong on many different levels  but most likely  the strings in your database are utf-8 encoded  and you are presenting them as iso-8859-1  Or the other way around  The proper way to fix this problem  is to get your character-sets straight  The simplest strategy  since you re using PHP  is to use iso-8859-1 throughout your application  To do this  you must ensure that   All PHP source-files are saved as iso-8859-1  Not to be confused with cp-1252   Your web-server is configured to serve files with charset iso-8859-1 Alternatively  you can override the webservers settings from within the PHP-document  using header  In addition  you may insert a meta-tag in you HTML  that specifies the same thing  but this isn t strictly needed  You may also specify the accept-charset attribute on your  lt form gt  elements  Database tables are defined with encoding as latin1 The database connection between PHP to and database is set to latin1  If you already have data in your database  you should be aware that they are probably messed up already  If you are not already in production phase  just wipe it all and start over  Otherwise you ll have to do some data cleanup  A note on meta-tags  since everybody misunderstands what they are  When a web-server serves a file  A HTML-document   it sends some information  that isn t presented directly in the browser  This is known as HTTP-headers  One such header  is the Content-Type header  which specifies the mimetype of the file  Eg  text html  as well as the encoding  aka charset   While most webservers will send a Content-Type header with charset info  it s optional  If it isn t present  the browser will instead interpret any meta-tags with http-equiv  quot Content-Type quot   It s important to realise that the meta-tag is only interpreted if the webserver doesn t send the header  In practice this means that it s only used if the page is saved to disk and then opened from there  This page has a very good explanation of these things

[php] PHP output showing little black diamonds with a question mark

Examples related to php

Examples related to encoding

Examples related to character-encoding