How to move a python 2.6 project to UTF-8?

We are moving from latin1 to UTF-8 and have 100k lines of python code.

Plus I'm new in python (ha-ha-ha!).

I already开发者_如何学运维 know that str() function fails when receiving Unicode so we should use unicode() instead of it with almost the same effect.

What are the other "dangerous" places of code?

Are there any basic guidelines/algorithms for moving to UTF-8? Can it be written an automatic 'code transformer'?

---

str and unicode are classes, not functions. When you call str(u'abcd') you are initialising a new string which takes 'abcd' as a variable. It just so happens that str() can be used to convert a string of any type to an ascii str.

Other areas to look out for are when reading from a file/input, or basically anything you get back as a string from a function that was not written for unicode.

Enjoy :)

---

Can it be written an automatic 'code transformer'? =)

No. str and unicode are two different types which have different purposes. You should not attempt to replace every occurrence of a byte string with a Unicode string, neither in Python 2 nor Python 3.

Continue to use byte strings for binary data. In particular anything you're writing to a file or network socket is bytes. And use Unicode strings for user-facing text.

In between there is a grey area of internal ASCII-character strings which could equally be bytes or Unicode. In Python 2 these are typically bytes, in Python 3 typically Unicode. In you are happy to limit your code to Python 2.6+, you can mark your definitely-bytes strings as b'' and bytes, your definitely-characters strings as u'' and unicode, and use '' and str for the “whatever the default type of string is” strings.

---

One way to quickly convert Python 2.x to have a default encoding of UTF-8 is to set the default encoding. This approach has its downsides--primarily that it changes the encoding for all libraries as well as your application, so use with caution. My company uses that technique in our production apps and it suits us well. It's also forward-compatible with Python 3, which has UTF-8 as the default encoding. You'll still have to change references of str() to unicode(), but you won't have to explicitly specify the encoding with .decode() and encode().