Report #1: Unicode support

[Dmitry Selyutin]

This is a report on progress in improving Unicode support in FreeBSD.

During the early period, I've been studying Unicode Technical Standard,
which describes how to implement Unicode Collation Algorithm.
I've tried to use the patch proposed by Konrad Jankowski, but it was a
rather unsuccessful attempt, since this patch predates xlocale support,
implemented by David Chisnall. The initial plan was to port collation
support from Apple's libc library, but we rejected this idea because of
poor code quality. Moreover, if we decided to use Apple's libc, we would
have broken the entire xlocale support.

Having lost a significant amount of time on Apple's libc and Konrad's
path, we've decided to implement collation from scratch according to
Unicode Normalization Algorithm.

One of the requirements for collation is the normalization of the string
before performing actual collation. C Standard Library lacks such
feature, so I started to implement it. This work is almost finished; the
FreeBSD's libc will have __strnorm_l(), __strnorm() and __wcsnorm()
functions. They have man pages and can be already used to normalize
ASCII, Latin-1 and Hangul strings. The last part is to implement
normalization of the other characters, which is usually done using
database lookup (usually Unicode data is stored in arrays, where each
array denotes single Multilingual Plane).
These functions are designed in the way that may allow to include them
in POSIX standard later under strnorm(), strnorm_l() and wcsnorm()
names. If _LIBC_UNICODE_ADDENDA macro is defined, they will be already
available under these names.

Unicode Standard is a bit difficult: sometimes Unicode Standard focuses
on details, paying little attention to the main part. However, I'm
planning to finish normalization algorithm in day or two and then
implement a collation algorithm.

We lost a significant time on trying to port Konrad's patch and Apple's
libc collation algorithm. Now we focus on the Unicode Standard directly;
that seems to be a better decision. The first step is to implement
collation algorithm in the canonical way, then to focus on its
improvements and testing.

I'd also like to thank my mentors, Pedro and David, who were (and are)
so kind to give me advice throughout my work. It's particularly valuable
since our task is not so simple as it may seem to be. :-)


-- 
With best regards,
Dmitry Selyutin