From bugmaster at FreeBSD.org  Mon Sep  7 11:07:00 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Sep  7 11:08:29 2009
Subject: Current problem reports assigned to freebsd-i18n@FreeBSD.org
Message-ID: <200909071106.n87B6xpr010240@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o conf/137870  i18n       [locale] en_DK needed
f conf/109367  i18n       [locale] UTF8 encoded locales and problem collating ac
f conf/91106   i18n       [locale] date definitions in pl_PL locale are wrong

3 problems total.

From edwin at mavetju.org  Tue Sep  8 05:35:02 2009
From: edwin at mavetju.org (Edwin Groothuis)
Date: Tue Sep  8 05:35:09 2009
Subject: WIP - share/{monet,msg,numeric,time}def
Message-ID: <20090908053451.GA4568@mavetju.org>

In the last couple of months I've spend some time with the data in
the share/{monet,msg,numeric,time}def directories and the data from
the CLDR (Common Locale Data Repository) project.

The biggest issues with the way the current data in the *def directories
is maintained is that it is partly high-ascii (specially for the
non-US-ASCII and non-ISO8859-{1,2,15} character maps) and partly
un-synchronized between the different character maps for the same
locale.

The first approach was to see if I could transform the data from
the CLDR project into the format the FreeBSD project wanted to have
it.  It taught me a lot about the data stored in the CLDR project,
but also that it isn't compatible enough to do it automatic.

The second approach, still happening now, is going much better:
Instead of storing the high-ascii and multiple charactermap
translations in the SCM, we have per locale one file with a proper
definition of the words and syntax used, which gets converted into
UTF-8 and which then gets transformed to the required charactermaps.


For example, the file share/msgdef/nl_NL.unicode:

    # yesexpr
    ^[<LATIN SMALL LETTER J><LATIN CAPITAL LETTER J><LATIN SMALL LETTER Y><LATIN CAPITAL LETTER Y>].*
    # noexpr
    ^[<LATIN SMALL LETTER N><LATIN CAPITAL LETTER N>].*
    # EOF

gets converted into nl_NL.UTF-8:

    # yesexpr
    ^[jJyY].*
    # noexpr
    ^[nN].*
    # EOF

and gets transformed into its ISO8859-1 and ISO8859-15 equivalents.
Since this is low-ascii it is a boring example, but the idea is
there.


What are currently show-stoppers?

- The conversion between .unicode and .UTF-8 is done via a Perl
  script and the CLDR database, which means that it won't be in the
  base system for now. So we need both the .unicode source files
  and the .UTF-8 files in the SCM system.

- There is no iconv in the base operating system yet. Gabor@ is in
  the process of porting citrus-iconv from NetBSD, but it isn't
  available yet. So we also need the converted charactermaps in the
  SCM for now. I have access to his iconv and will feed the current
  issues back to him.


These two show-stoppers right now cause that we will get a lot more
data in the SCM system than what we have right now until they are
resolved. The first one should not be difficult, the second one is
with somebody who understands it :-)


So the advantages, when everything is ready:

- Human readable source files with Unicode style encoding.
- All locales with the different character maps are generated from
  one source and thus up-to-date with each other.


Once this part is working properly (and to others people satisfaction)
we can update the contents with information from third party sources
like the CLDR. But that is still a long time away for now.


Edwin
svn://svn.freebsd.org/base/user/edwin/locale

-- 
Edwin Groothuis		Website: http://www.mavetju.org/
edwin@mavetju.org	Weblog:  http://www.mavetju.org/weblog/
From bugmaster at FreeBSD.org  Mon Sep 14 11:07:01 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Sep 14 11:08:15 2009
Subject: Current problem reports assigned to freebsd-i18n@FreeBSD.org
Message-ID: <200909141107.n8EB70Do072341@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o conf/137870  i18n       [locale] en_DK needed
f conf/109367  i18n       [locale] UTF8 encoded locales and problem collating ac
f conf/91106   i18n       [locale] date definitions in pl_PL locale are wrong

3 problems total.

From edwin at mavetju.org  Wed Sep 16 21:50:23 2009
From: edwin at mavetju.org (Edwin Groothuis)
Date: Wed Sep 16 21:50:30 2009
Subject: WIP - share/{monet,msg,numeric,time}def
In-Reply-To: <20090908053451.GA4568@mavetju.org>
Message-ID: <20090916214956.GA95326@mavetju.org>

> Once this part is working properly (and to others people satisfaction)

Tumbleweed all over the place!

Right now the output of this WIP matches the FreeBSD datafiles.

So if somebody smart wants to give his 2cents on this, please do now.
And not after I merged the data into head please :-)

Is there somebody here who understands the hi_IN language (.ISCII-DEV
charactermap) and the ru_RU language (CP866 charactermap) for some
questions?

Edwin

-- 
Edwin Groothuis		Website: http://www.mavetju.org/
edwin@mavetju.org	Weblog:  http://www.mavetju.org/weblog/
From snezhko at indorsoft.ru  Thu Sep 17 03:00:16 2009
From: snezhko at indorsoft.ru (Victor Snezhko)
Date: Thu Sep 17 03:00:22 2009
Subject: WIP - share/{monet,msg,numeric,time}def
In-Reply-To: <20090916214956.GA95326@mavetju.org> (Edwin Groothuis's message
	of "Thu, 17 Sep 2009 07:49:56 +1000")
References: <20090916214956.GA95326@mavetju.org>
Message-ID: <ufxamgunr.fsf@indorsoft.ru>

Edwin Groothuis <edwin@mavetju.org> writes:

> Is there somebody here who understands the hi_IN language (.ISCII-DEV
> charactermap) and the ru_RU language (CP866 charactermap) for some
> questions?

Yes for ru_RU. Not quite familiar with locale infrastructure though.

-- 
Victor Snezhko



From ache at nagual.pp.ru  Fri Sep 18 20:33:21 2009
From: ache at nagual.pp.ru (Andrey Chernov)
Date: Fri Sep 18 20:33:28 2009
Subject: WIP - share/{monet,msg,numeric,time}def
In-Reply-To: <20090916214956.GA95326@mavetju.org>
References: <20090908053451.GA4568@mavetju.org>
	<20090916214956.GA95326@mavetju.org>
Message-ID: <20090918202305.GA9899@nagual.pp.ru>

On Thu, Sep 17, 2009 at 07:49:56AM +1000, Edwin Groothuis wrote:
> > Once this part is working properly (and to others people satisfaction)
> So if somebody smart wants to give his 2cents on this, please do now.
> And not after I merged the data into head please :-)

Please don't forget about recently added lt_LT.

> Is there somebody here who understands the hi_IN language (.ISCII-DEV
> charactermap) and the ru_RU language (CP866 charactermap) for some
> questions?

ru_RU.CP866 yes.

-- 
http://ache.pp.ru/
From bugmaster at FreeBSD.org  Mon Sep 21 11:06:56 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Sep 21 11:08:18 2009
Subject: Current problem reports assigned to freebsd-i18n@FreeBSD.org
Message-ID: <200909211106.n8LB6tBU030264@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o conf/137870  i18n       [locale] en_DK needed
f conf/109367  i18n       [locale] UTF8 encoded locales and problem collating ac
f conf/91106   i18n       [locale] date definitions in pl_PL locale are wrong

3 problems total.

From bugmaster at FreeBSD.org  Mon Sep 28 11:06:55 2009
From: bugmaster at FreeBSD.org (FreeBSD bugmaster)
Date: Mon Sep 28 11:08:13 2009
Subject: Current problem reports assigned to freebsd-i18n@FreeBSD.org
Message-ID: <200909281106.n8SB6tRg064020@freefall.freebsd.org>

Note: to view an individual PR, use:
  http://www.freebsd.org/cgi/query-pr.cgi?pr=(number).

The following is a listing of current problems submitted by FreeBSD users.
These represent problem reports covering all versions including
experimental development code and obsolete releases.


S Tracker      Resp.      Description
--------------------------------------------------------------------------------
o conf/137870  i18n       [locale] en_DK needed
f conf/109367  i18n       [locale] UTF8 encoded locales and problem collating ac
f conf/91106   i18n       [locale] date definitions in pl_PL locale are wrong

3 problems total.