MySQL PHP & UTF-8

Its been a while but I was getting characters like £ instead of £ (for example) and it took a quick refresher to realise what was going on. A fantastic background reading is available here.

So what is going on?
UTF-8 uses one or more 8-bit bytes to store a single character, whereas ASCII for example uses only one byte per character. This makes it more space-efficient than UTF-16 or UTF-32 as the majority of characters can be single byte encoded but with the added benefit that you can store any character, – should you need to. It uses the most significant bits of each byte as continuation bits (to signify that the following byte(s) form part of the same character) and it this is the reason that improperly displayed UTF-8 results in weird characters.

OK, enough history, – lets just fix it!
The solution is to make sure you set the character set correctly with a Content-type header.

Content-type: text/html; charset=utf-8
<meta http-equiv="content-type" content="text/html; charset=utf-8">

So now that we are ready to store UTF-8 data we store some rows of data and output to the browser looks fine. However, is it really?

MySQL needs to know the character set of the data sent to it in your queries. The default connection character set is ISO 8859-1 (latin1) — treating all your supplied data as if it was ISO 8859-1, which is then automatically converted to the underlying character set of the column (UTF-8 in our case). Taking our earlier example this means that the two-byte pound sign is perceived as two ISO 8859-1 characters: Â and £. MySQL will then encode these characters separately as UTF-8 requiring 2 bytes each – double encoding as we use 4 bytes to store a single pound character! When selecting data from the table, the reverse occurs when MySQL converts the UTF-8 back into ISO 8859-1 and the browser (correctly) interprets the two bytes as a pound sign. The problem here is that while everything looks correct it is needlessly using extra storage space and CPU cycles in the conversion.

It’s very simple to change the connection character set and avoid these overheads. The query “SET NAMES ‘utf8′” instructs MySQL to treat incoming data as UTF-8, which can be directly inserted into columns with a matching charset. In practice you may issue this query after initialising your connection to the MySQL server so all future communication will be in UTF-8.

One major problem occurs when you have a table full of double-encoded UTF-8 text because it was inserted before you knew about changing the connection character set. If you then add in the extra “SET NAMES” query and output the retrieved text to a browser you will notice all the extra characters have crept in.

You may be tempted to be selective about where you use SET NAMES ‘utf8’ however it is fairly simple to modify all of the data in place and end up with clean tables. Here is some pseudocode that pulls valid UTF-8 from the table and reinserts it after switching the connection charset:

SET NAMES 'latin1';
SELECT id, col1, col2, col3, col4 FROM table;
SET NAMES 'utf8';
for each row
INSERT INTO table (col1, col2, col3, col4) VALUES (, , , ) WHERE id = ;