UTF8 characters not printed as such in Drupals HTML

I am trying to debug a nasty utf-8 problem, and do not know where to start.

A page contains the word 'categorieën', wich should be categorieën. Clearly something is wrong with the UTF-8. This happens with all these multibite characters. I have scanned the gazillion topics here on UTF8, but they mostly cover the basics, not this situation where everything appears to be configured and set correct, but clearl开发者_Python百科y is not.

The pages are served by Drupal, from a MySQL database.

The database was migrated (not by me) by sql-dumping and -importing trough phpmyadmin. Good chance something went wrong there, because before, there was no problem. And because the problem occurs only on older, imported items. Editing these items or inserting new ones, and fixxing the wrongly encoded characters by hand, fixes the problem. Though I cannot see a difference in the database.

• Content re-edited trough Drupal does not have this problem.
• When, on the CLI, using MySQL, I can read out that text and get the correct ë character. On both The articles that render "correct "and "incorrect" characters.
• The tables have collation utf8_general_ci
• Headers appear to be sent with correct encoding: Vary Accept-Encoding and Content-Type text/html; charset=utf-8
• HTML head contains a <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
• the HTTP headers tell me there is a Varnish proxy inbetween. Could that cause UTF8-conversion/breakage
• content is served Gzipped, normal in Drupal, and I have never seen this UTF8 issie wrt the gzipping, but you never know.

It appears the import is the culprit and I would like to know a) what went wrong. b) why I cannot see a difference in the mysql cli client between "wrong" and "correct" characters c) how to fix the database, or where to start looking and learning on how to fix it.

---

The dump file was probably output as UTF-8, but interpreted as latin1 during import.

The ë, the latin1 two-byte representation of UTF-8's ë, is physically in your tables as UTF-8 data.

Seeing as you have a mix of intact and broken data, this will be tough to fix in a general way, but usually, this dirty workaround^* will work well:

UPDATE table SET column = REPLACE("ë", "ë", column);

Unless you are working with languages other than dutch, the range of broken characters should be extremely limited and you might be able to fix it with a small number of such statements.

Related questions with the same problem:

• Detecting utf8 broken characters in MySQL
• I need help fixing Broken UTF8 encoding

_{* (of course, don't forget to make backups before running anything like this!)}

---

There should have not gone anything awol in exporting and importing a Drupal dump, unless the person doing this somehow succeeded into setting the export as something else than UTF8. We export/import dumps a lot and have never bumped into a such problem.

Hopefully Pekkas answers will help you to resolve the issue, if it is in the DB, but I also thought that you could check wether the data being shown on the web page is being ran through some php functions that arent multibyte friendly.

Here are some equivalents of normal functions in mb: http://php.net/manual/en/ref.mbstring.php

ps. If you have recently moved your site to another server (so it's not just a db import), you should check what headers your site is sending out with a tool such as http://www.webconfs.com/http-header-check.php

Make sure the last row has UTF8 in it.

---

You mention that the import might be the problem. In that case it's possible that during import the connection with the client and the MySQL server wasn't using UTF-8. I've had this problem a couple of times in the past, so I'd like to share with you these MySQL settings (in my.conf):

Under the server settings add these:

# UTF 8
default-character-set=utf8
character-set-server=utf8
collation-server=utf8_general_ci
skip-character-set-client-handshake

And under the client settings add:

default-character-set=utf8

This might save you some headache the next time.

---

To be absolutely sure you have utf8 from start to end:
- source code files in utf8 without BOM
- database with utf8 collation
- database tables with utf8 collation
- database connection in utf8 (query it with 'SET CHARSET UTF8')
- pages header set to utf8 (the ajax ones too) - meta tag to set page in utf8