How do I get unicode support in python?

[Heiko Wundram (Beenic)]

Am Freitag, 8. Februar 2008 15:26:48 schrieb Eric Mesa:
> I'm running a web server with FreeBSD 6.1-RELEASE and python 2.4.3.  I'm
> unable to print any characters outside of ascii.  I have tried this code on
> my Linux computer, which has python 2.5.x and it works - so the code is
> solid.
>
> What do I need to do to get python on the web server to have unicode
> support?  Is there a module/package I need to import in the 2.4 series?  Or
> is there some package/port I need to install?  Or do I just recompile
> python with some different flags?  (And does that entail any uninstalling
> first?)

For Python to be able to "print" unicode characters to the console, it must 
know the encoding of the console. Generally, this entails setting up LC_ALL 
and LANG and of course your terminal (emulator) appropriately, and testing 
whether the interpreter sets the correct encoding on startup (which can be 
found as sys.getdefaultencoding()). When the encoding that the interpreter 
uses to "print" _unicode_-strings cannot encode the unicode characters you 
hand it to the current default encoding, the codec barfs:

[modelnine at phoenix ~]$ python
Python 2.5.1 (r251:54863, Nov  6 2007, 19:02:51)
[GCC 4.2.1 20070719  [FreeBSD]] on freebsd7
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> print u"\xfa"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfa' in position 
0: ordinal not in range(128)
>>> print u"\xfa".encode("latin-1")
ú
>>>

Basically, the easiest resolution is to do the conversion yourself (like I did 
in the second example). The other possibility is to change the deault 
encoding to something that matches your default console (probably latin-1), 
which you can do in /usr/local/lib/python2x/site.py.

HTH!

-- 
Heiko Wundram
Product & Application Development