gnu/116363: isspace broken for UTF-8 locales

[Hye-Shik Chang]

The following reply was made to PR gnu/116363; it has been noted by GNATS.

From: Hye-Shik Chang <perky at FreeBSD.ORG>
To: Andrey Chernov <ache at nagual.pp.ru>, Petr Hroudny <petr.hroudny at gmail.com>,
        freebsd-gnats-submit at FreeBSD.ORG, jkoshy at FreeBSD.ORG, i18n at FreeBSD.ORG
Cc:  
Subject: Re: gnu/116363: isspace broken for UTF-8 locales
Date: Mon, 17 Sep 2007 01:22:14 +0900

 On Sun, Sep 16, 2007 at 12:54:33PM +0400, Andrey Chernov wrote:
 > On Sat, Sep 15, 2007 at 09:08:01AM +0000, Petr Hroudny wrote:
 > > 
 > > >Number:         116363
 > > >Category:       gnu
 > > >Synopsis:       isspace broken for UTF-8 locales
 > > >Confidential:   no
 > > >Severity:       non-critical
 > > >Priority:       medium
 > > >Responsible:    freebsd-bugs
 > > >State:          open
 > > >Quarter:        
 > > >Keywords:       
 > > >Date-Required:
 > > >Class:          sw-bug
 > > >Submitter-Id:   current-users
 > > >Arrival-Date:   Sat Sep 15 09:10:02 GMT 2007
 > > >Closed-Date:
 > > >Last-Modified:
 > > >Originator:     Petr Hroudny
 > > >Release:        6-stable, 7-current
 > > >Organization:
 > > >Environment:
 > > >Description:
 > > In UTF-8 locales, isspace(0xA0) returns 1 which is wrong.
 > > 
 > > In UTF-8, 0xA0 could only be the second or third byte of multibyte character, but never a space.
 > > 
 > > As a consequence, operations like str.upper() and/or str.split() are broken, when
 > > UTF-8 character with 0xA0 byte is encountered.
 
 If you are saying about Python's str.split(), the problem is due
 to our libc bug (or feature) which is described many times before,
 and Python already includes a workaround for the problem.
 http://mail.python.org/pipermail/python-checkins/2004-August/042343.html
 
 > It seems that our UTF-8.src is completely wrong, it is just plain Unicode 
 > and not UTF-8 which multibyte values should start from
 > C2-DF
 > E0-EF
 > F0-F4
 > only (as stated in http://en.wikipedia.org/wiki/UTF-8 f.e.)
 > Can anybody write replacement for it?
 
 In fact, UTF-8.src defines values for not UTF-8 but Unicode codepoints.
 Using the Unicode codepoint as wchar_t's internal representation gives
 much benefit.  I think we would be better to make isspace() and
 other ctypes functions aware of "encoding".  IIRC, tjr@ provided the
 workaround as in the URL mentioned above and said that it would get
 a chance to be fixed in 6 or 7 on 2004.
 
 Hye-Shik