I have a backup server that automatically backs up my live site, both files and database.
On the live site, the text looks fine, but when you view the mirrored version of it, it displays '?' within some of the text. This text is stored within the news database table.
Here is a screen shot of it being on the live server and of it on the mirrored server.
What could happen within the process of backing it up to the mirrored server?
This question is related to
html
backup
character-encoding
mirror
The answer is
Check the character set being emitted by your mirrored server. There appears to be a difference from that to the main server -- the live site appears to be outputting Unicode, where the mirror is not. Also, it's usually a good idea to scrub Unicode characters in your incoming content and replace them with their appropriate HTML entities.
Your specific issue regards "smart quotes," "em dashes" and "en dashes." I know you can replace em dashes with — and n-dashes with – (which should be done on the input side of your database); I don't know what the correct replacement for the smart quotes would be. (I usually just replace all curly single quotes with ' and all curly double quotes with " ... Typography geeks may feel free to shoot me on sight.)
I should note that some browsers are more forgiving than others with this issue -- Internet Explorer on Windows tends to auto-magically detect and "fix" this; Firefox and most other browsers display the question marks.
This is going to be something to do with character encodings.
Are you sure the mirrored site has the same properties with regards to character encodings as your main server?
Depending on what sort of server you have, this may be a property of the server process itself, or it could be an environment variable.
For example, if this is a UNIX environment, perhaps try comparing LANG or LC_ALL?
See also here
I usually curse MS word and then run the following Wscript.
// replace with path to a file that needs cleaning
PATH = "test.html"
var go=WScript.CreateObject("Scripting.FileSystemObject");
var content=go.GetFile(PATH).OpenAsTextStream().ReadAll();
var out=go.CreateTextFile("clean-"+PATH, true);
// symbols
content=content.replace(/“/g,'"');
content=content.replace(/”/g,'"');
content=content.replace(/’/g,"'");
content=content.replace(/–/g,"-");
content=content.replace(/©/g,"©");
content=content.replace(/®/g,"®");
content=content.replace(/°/g,"°");
content=content.replace(/¶/g,"<p>");
content=content.replace(/¿/g,"¿");
content=content.replace(/¡/g,'¡');
content=content.replace(/¢/g,'¢');
content=content.replace(/£/g,'£');
content=content.replace(/¥/g,'¥');
out.Write(content);
Your browser hasn't interpretted the encoding of the page correctly (either because you've forced it to a particular setting, or the page is set incorrectly), and thus cannot display some of the characters.
Edit your Apache configuration file on the "mirror" server (the server with the problem), and comment-out the following line:
AddDefaultCharset UTF-8
Then restart Apache:
service httpd restart
The problem is that the "AddDefaultCharset UTF-8" line overrides the Content-Type specified in the .html files; e.g.:
<meta http-equiv=Content-Type content="text/html; charset=windows-1252">
The most common symptom is that character codes above 127 display as black diamonds with question marks on them (in Chrome, Safari or Firefox), or as little boxes (in IE and Opera). HTML files generated by Microsoft Word usually have many such characters, the most common one being character code 160 = 0xA0, which is equivalent to " " in the Windows-1252 encoding, and is often found between span tags, like this:
<span style="mso-spacerun: yes">ááá </span>
Unicode or other character set characters falling through?
I have seen similar "strange" characters show up on sites I have worked on often when the text is copied from an email or some other document format (e.g. word) into a text editor. The editor can display the non ASCII characters but the browser can't. For the website, I would suggest looking up the HTML entity code for the character and inserting that instead ... or switch to more standard ones.
I got here looking for a solution for JavaScript displayed in the browser and although not directly related with a database...
In my case I copied and pasted some text I found on the internet into a JavaScript file and saved it with Windows Notepad.
When the page that uses that JavaScript file output the strings there were question marks (like the ones shown in the question) instead of the special characters like accented letters, etc.
I opened the file using Notepad++. Right after opening the file I saw that the character encoding was set as ANSI as you can see (mouse cursor on footer) in the following screenshot:
To solve the issue, click the Encoding menu in Notepad++ and select Encode in UTF-8. You should be good to go. :)
Your browser hasn't interpretted the encoding of the page correctly (either because you've forced it to a particular setting, or the page is set incorrectly), and thus cannot display some of the characters.
This is going to be something to do with character encodings.
Are you sure the mirrored site has the same properties with regards to character encodings as your main server?
Depending on what sort of server you have, this may be a property of the server process itself, or it could be an environment variable.
For example, if this is a UNIX environment, perhaps try comparing LANG or LC_ALL?
See also here
I had this issue so I just took all my content, copy/pasted it into notepad, made a new php file, pasted back in, re-saved and overwrote, and.. that worked! It really was some relic of Microsoft Word editing...
Unicode or other character set characters falling through?
I have seen similar "strange" characters show up on sites I have worked on often when the text is copied from an email or some other document format (e.g. word) into a text editor. The editor can display the non ASCII characters but the browser can't. For the website, I would suggest looking up the HTML entity code for the character and inserting that instead ... or switch to more standard ones.
Your browser hasn't interpretted the encoding of the page correctly (either because you've forced it to a particular setting, or the page is set incorrectly), and thus cannot display some of the characters.
I usually curse MS word and then run the following Wscript.
// replace with path to a file that needs cleaning
PATH = "test.html"
var go=WScript.CreateObject("Scripting.FileSystemObject");
var content=go.GetFile(PATH).OpenAsTextStream().ReadAll();
var out=go.CreateTextFile("clean-"+PATH, true);
// symbols
content=content.replace(/“/g,'"');
content=content.replace(/”/g,'"');
content=content.replace(/’/g,"'");
content=content.replace(/–/g,"-");
content=content.replace(/©/g,"©");
content=content.replace(/®/g,"®");
content=content.replace(/°/g,"°");
content=content.replace(/¶/g,"<p>");
content=content.replace(/¿/g,"¿");
content=content.replace(/¡/g,'¡');
content=content.replace(/¢/g,'¢');
content=content.replace(/£/g,'£');
content=content.replace(/¥/g,'¥');
out.Write(content);
Unicode or other character set characters falling through?
I have seen similar "strange" characters show up on sites I have worked on often when the text is copied from an email or some other document format (e.g. word) into a text editor. The editor can display the non ASCII characters but the browser can't. For the website, I would suggest looking up the HTML entity code for the character and inserting that instead ... or switch to more standard ones.
I usually curse MS word and then run the following Wscript.
// replace with path to a file that needs cleaning
PATH = "test.html"
var go=WScript.CreateObject("Scripting.FileSystemObject");
var content=go.GetFile(PATH).OpenAsTextStream().ReadAll();
var out=go.CreateTextFile("clean-"+PATH, true);
// symbols
content=content.replace(/“/g,'"');
content=content.replace(/”/g,'"');
content=content.replace(/’/g,"'");
content=content.replace(/–/g,"-");
content=content.replace(/©/g,"©");
content=content.replace(/®/g,"®");
content=content.replace(/°/g,"°");
content=content.replace(/¶/g,"<p>");
content=content.replace(/¿/g,"¿");
content=content.replace(/¡/g,'¡');
content=content.replace(/¢/g,'¢');
content=content.replace(/£/g,'£');
content=content.replace(/¥/g,'¥');
out.Write(content);
Check the character set being emitted by your mirrored server. There appears to be a difference from that to the main server -- the live site appears to be outputting Unicode, where the mirror is not. Also, it's usually a good idea to scrub Unicode characters in your incoming content and replace them with their appropriate HTML entities.
Your specific issue regards "smart quotes," "em dashes" and "en dashes." I know you can replace em dashes with — and n-dashes with – (which should be done on the input side of your database); I don't know what the correct replacement for the smart quotes would be. (I usually just replace all curly single quotes with ' and all curly double quotes with " ... Typography geeks may feel free to shoot me on sight.)
I should note that some browsers are more forgiving than others with this issue -- Internet Explorer on Windows tends to auto-magically detect and "fix" this; Firefox and most other browsers display the question marks.
Your browser hasn't interpretted the encoding of the page correctly (either because you've forced it to a particular setting, or the page is set incorrectly), and thus cannot display some of the characters.
Edit your Apache configuration file on the "mirror" server (the server with the problem), and comment-out the following line:
AddDefaultCharset UTF-8
Then restart Apache:
service httpd restart
The problem is that the "AddDefaultCharset UTF-8" line overrides the Content-Type specified in the .html files; e.g.:
<meta http-equiv=Content-Type content="text/html; charset=windows-1252">
The most common symptom is that character codes above 127 display as black diamonds with question marks on them (in Chrome, Safari or Firefox), or as little boxes (in IE and Opera). HTML files generated by Microsoft Word usually have many such characters, the most common one being character code 160 = 0xA0, which is equivalent to " " in the Windows-1252 encoding, and is often found between span tags, like this:
<span style="mso-spacerun: yes">ááá </span>
Check the character set being emitted by your mirrored server. There appears to be a difference from that to the main server -- the live site appears to be outputting Unicode, where the mirror is not. Also, it's usually a good idea to scrub Unicode characters in your incoming content and replace them with their appropriate HTML entities.
Your specific issue regards "smart quotes," "em dashes" and "en dashes." I know you can replace em dashes with — and n-dashes with – (which should be done on the input side of your database); I don't know what the correct replacement for the smart quotes would be. (I usually just replace all curly single quotes with ' and all curly double quotes with " ... Typography geeks may feel free to shoot me on sight.)
I should note that some browsers are more forgiving than others with this issue -- Internet Explorer on Windows tends to auto-magically detect and "fix" this; Firefox and most other browsers display the question marks.
I had this issue so I just took all my content, copy/pasted it into notepad, made a new php file, pasted back in, re-saved and overwrote, and.. that worked! It really was some relic of Microsoft Word editing...
Check the character set being emitted by your mirrored server. There appears to be a difference from that to the main server -- the live site appears to be outputting Unicode, where the mirror is not. Also, it's usually a good idea to scrub Unicode characters in your incoming content and replace them with their appropriate HTML entities.
Your specific issue regards "smart quotes," "em dashes" and "en dashes." I know you can replace em dashes with — and n-dashes with – (which should be done on the input side of your database); I don't know what the correct replacement for the smart quotes would be. (I usually just replace all curly single quotes with ' and all curly double quotes with " ... Typography geeks may feel free to shoot me on sight.)
I should note that some browsers are more forgiving than others with this issue -- Internet Explorer on Windows tends to auto-magically detect and "fix" this; Firefox and most other browsers display the question marks.
I got here looking for a solution for JavaScript displayed in the browser and although not directly related with a database...
In my case I copied and pasted some text I found on the internet into a JavaScript file and saved it with Windows Notepad.
When the page that uses that JavaScript file output the strings there were question marks (like the ones shown in the question) instead of the special characters like accented letters, etc.
I opened the file using Notepad++. Right after opening the file I saw that the character encoding was set as ANSI as you can see (mouse cursor on footer) in the following screenshot:
To solve the issue, click the Encoding menu in Notepad++ and select Encode in UTF-8. You should be good to go. :)
This is going to be something to do with character encodings.
Are you sure the mirrored site has the same properties with regards to character encodings as your main server?
Depending on what sort of server you have, this may be a property of the server process itself, or it could be an environment variable.
For example, if this is a UNIX environment, perhaps try comparing LANG or LC_ALL?
See also here
I usually curse MS word and then run the following Wscript.
// replace with path to a file that needs cleaning
PATH = "test.html"
var go=WScript.CreateObject("Scripting.FileSystemObject");
var content=go.GetFile(PATH).OpenAsTextStream().ReadAll();
var out=go.CreateTextFile("clean-"+PATH, true);
// symbols
content=content.replace(/“/g,'"');
content=content.replace(/”/g,'"');
content=content.replace(/’/g,"'");
content=content.replace(/–/g,"-");
content=content.replace(/©/g,"©");
content=content.replace(/®/g,"®");
content=content.replace(/°/g,"°");
content=content.replace(/¶/g,"<p>");
content=content.replace(/¿/g,"¿");
content=content.replace(/¡/g,'¡');
content=content.replace(/¢/g,'¢');
content=content.replace(/£/g,'£');
content=content.replace(/¥/g,'¥');
out.Write(content);
Source: Stackoverflow.com