gnu/116363: isspace broken for UTF-8 locales

[Hye-Shik Chang]

On Sun, Sep 16, 2007 at 12:54:33PM +0400, Andrey Chernov wrote:
> On Sat, Sep 15, 2007 at 09:08:01AM +0000, Petr Hroudny wrote:
> > 
> > >Number:         116363
> > >Category:       gnu
> > >Synopsis:       isspace broken for UTF-8 locales
> > >Confidential:   no
> > >Severity:       non-critical
> > >Priority:       medium
> > >Responsible:    freebsd-bugs
> > >State:          open
> > >Quarter:        
> > >Keywords:       
> > >Date-Required:
> > >Class:          sw-bug
> > >Submitter-Id:   current-users
> > >Arrival-Date:   Sat Sep 15 09:10:02 GMT 2007
> > >Closed-Date:
> > >Last-Modified:
> > >Originator:     Petr Hroudny
> > >Release:        6-stable, 7-current
> > >Organization:
> > >Environment:
> > >Description:
> > In UTF-8 locales, isspace(0xA0) returns 1 which is wrong.
> > 
> > In UTF-8, 0xA0 could only be the second or third byte of multibyte character, but never a space.
> > 
> > As a consequence, operations like str.upper() and/or str.split() are broken, when
> > UTF-8 character with 0xA0 byte is encountered.

If you are saying about Python's str.split(), the problem is due
to our libc bug (or feature) which is described many times before,
and Python already includes a workaround for the problem.
http://mail.python.org/pipermail/python-checkins/2004-August/042343.html

> It seems that our UTF-8.src is completely wrong, it is just plain Unicode 
> and not UTF-8 which multibyte values should start from
> C2-DF
> E0-EF
> F0-F4
> only (as stated in http://en.wikipedia.org/wiki/UTF-8 f.e.)
> Can anybody write replacement for it?

In fact, UTF-8.src defines values for not UTF-8 but Unicode codepoints.
Using the Unicode codepoint as wchar_t's internal representation gives
much benefit.  I think we would be better to make isspace() and
other ctypes functions aware of "encoding".  IIRC, tjr@ provided the
workaround as in the URL mentioned above and said that it would get
a chance to be fixed in 6 or 7 on 2004.

Hye-Shik