Report #1: Unicode support

[Pedro Giffuni]

Hi Dmitry and list;

On 06/15/14 17:25, Dmitry Selyutin wrote:
> This is a report on progress in improving Unicode support in FreeBSD.
>
> During the early period, I've been studying Unicode Technical Standard,
> which describes how to implement Unicode Collation Algorithm.
> I've tried to use the patch proposed by Konrad Jankowski, but it was a
> rather unsuccessful attempt, since this patch predates xlocale support,
> implemented by David Chisnall. The initial plan was to port collation
> support from Apple's libc library, but we rejected this idea because of
> poor code quality. Moreover, if we decided to use Apple's libc, we would
> have broken the entire xlocale support.
>
> Having lost a significant amount of time on Apple's libc and Konrad's
> path, we've decided to implement collation from scratch according to
> Unicode Normalization Algorithm.

I wouldn't call it exactly a "waste of time". I think it was important to
rescue the work done by the previous Summer of Code plus Dmitry
had to learn his way around FreeBSD's libc. Of course the code has
changed a lot and the approach was not really successful but it was
useful nevertheless. An important part of this work will be testing
and Konrad did set up a set of tests.

> One of the requirements for collation is the normalization of the string
> before performing actual collation. C Standard Library lacks such
> feature, so I started to implement it. This work is almost finished; the
> FreeBSD's libc will have __strnorm_l(), __strnorm() and __wcsnorm()
> functions. They have man pages and can be already used to normalize
> ASCII, Latin-1 and Hangul strings. The last part is to implement
> normalization of the other characters, which is usually done using
> database lookup (usually Unicode data is stored in arrays, where each
> array denotes single Multilingual Plane).
> These functions are designed in the way that may allow to include them
> in POSIX standard later under strnorm(), strnorm_l() and wcsnorm()
> names. If _LIBC_UNICODE_ADDENDA macro is defined, they will be already
> available under these names.
>
> Unicode Standard is a bit difficult: sometimes Unicode Standard focuses
> on details, paying little attention to the main part. However, I'm
> planning to finish normalization algorithm in day or two and then
> implement a collation algorithm.
>
> We lost a significant time on trying to port Konrad's patch and Apple's
> libc collation algorithm. Now we focus on the Unicode Standard directly;
> that seems to be a better decision. The first step is to implement
> collation algorithm in the canonical way, then to focus on its
> improvements and testing.
>
> I'd also like to thank my mentors, Pedro and David, who were (and are)
> so kind to give me advice throughout my work. It's particularly valuable
> since our task is not so simple as it may seem to be. :-)
>
>

It is indeed a difficult task and Dmitry will be very busy these days ;).

Pedro.