From mihai.carabas at gmail.com Fri Aug 1 15:42:38 2014 From: mihai.carabas at gmail.com (Mihai Carabas) Date: Fri, 1 Aug 2014 18:42:37 +0300 Subject: [GSOC] bhyve instruction caching In-Reply-To: References: Message-ID: Hi, Until now I managed to finish up all the coding stuff related to instruction caching. As you saw in my previous e-mails we obtained a speed-up of 35%-40% in the microbenchmarking tests (accessing LAPIC many times from a kernel module). Further we wanted to see how get this extrapolated to real-world workloads. I've made two kinds of benchmarking: a CPU intensive process and a make buildworld -j2 command. For each of one I've measured the time spent to execute. 1) The CPU intensive app is a bash script: #!/usr/local/bin/bash a=0 MAX=10000000 for i in $(seq 1 $MAX); do a=$((a+1)) done For a VM with 2 vCPUs: *Cache_instr=1 real 3m45.067s 3m42.628s 3m38.371s 3m36.301s 3m39.929s user 3m10.454s 3m8.785s 3m7.516s 3m8.204s 3m8.822s sys 0m19.085s 0m16.135s 0m13.696s 0m13.016s 0m16.105s * Cache_instru=0 real 3m50.550s 3m41.517s 3m34.783s user 3m5.350s 3m7.571s 3m1.415s sys 0m25.268s 0m19.200s 0m16.200s There are multiple measurements. As you can see the results aren't stable and are in the same range. To minimize the range they vary, I repeated the tests with 1vCPU (to eliminate the context switches): With 1vCPU: * Cache_instr=1 real 2m58.968s 2m57.009s 3m0.451s 2m55.902s 2m56.422s user 2m46.909s 2m45.241s 2m45.670s 2m45.788s 2m45.503s sys 0m4.890s 0m4.134s 0m3.942s 0m3.764s 0m3.984s * Cache_instr=0 real 2m56.845s 2m57.051s 3m1.794s 2m57.340s user 2m45.232s 2m44.873s 2m45.482s 2m46.538s sys 0m4.644s 0m4.141s 0m3.906s 0m3.875s As you can see the results are very appropiate in terms of variation and almost the same. 2) For a make buildworld -j2 with 1 vCPU: Cache_instr=1 13900.60 real 12051.54 user 1800.42 sys Cache_instr=0 13938.07 real 12122.14 user 1743.61 sys As you can see the difference between them is not significant and is about the same. As you can see for this two different kind of workloads there is no speed-up improvement unfortunatelly. I've tried other workloads more speific like: a) dd if=/dev/zero of=/dev/zero bs=256 count=10000K (from memory to memory - to not be influenced by the storage system) b) A simple getuid program that executes getuid syscall in a loop: int main(int argc, char *argv[]) { int i; if (argc == 2) { i = atoi(argv[1]); } else { i = 100; } while (i > 0) { getuid(); i--; } return 0; } But the results were the same. I spoke with Neel and it seems that we can't get a real-world benefict with this instruction caching. Thanks, Mihai From ghostman.sd at gmail.com Sat Aug 2 01:18:34 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Sat, 02 Aug 2014 05:17:53 +0400 Subject: Report #5 Message-ID: <53DC3C41.7070105@gmail.com> Hello everyone! Here is my report on progress that was achieved during this time. I've implemented actual Unicode Collation Algorithm for DUCET (Default Unicode Collation Element Table). I had to rewrite the entire implementation: I wasn't satisfied with its quality and the way that I've organized my source code, so I reverted my code and started again. My previous implementation was full of hard-coded parts and it was a bit harder to take anything useful from it for any other project. Now the entire implementation is available in include/unicode.h and lib/libc/unicode. If macro _UNICODE_SOURCE is defined, then wcscoll() will use new collation algorithm. struct _xlocale was modified in the way it will use two new members, colltable and collsize, which are just transmitted to __ucscoll(). If element is not found in the given table or table is NULL, then __ucscoll() tries to find this element in DUCET; if element was not found, then __ucscoll generates collation. I couldn't understand how the alternate shall be used though; it seems that it can be dropped since wcscoll() doesn't has any version that supports tailoring. I left it for now, but I'm pretty sure that we can omit it. I hadn't time to test wcscoll() better (especially using files provided by Unicode Character Database), so this is the task that I will do right now. :-) There are still several ways to improve the speed of the algorithm, but I feel that the time for it hasn't come yet. style(9) issues will also be handled (if any), just too tired to do it right now. __ucscoll() just uses __ucsxfrm(), then compares the strings using wcscmp() (this is the only platform-dependent part of code, I was too lazy to write __ucslen(), so I left it as it is). This collation algorithm support three levels; the last IIRC is usually the character itself if not defined, so I decided to omit it (especially since I'm not sure how variable weights should be handled). Any thoughs? -- With best regards, Dmitry Selyutin From ghostman.sd at gmail.com Sat Aug 2 01:23:19 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Sat, 02 Aug 2014 05:22:38 +0400 Subject: Report #5: Unicode support In-Reply-To: <53DC3C41.7070105@gmail.com> References: <53DC3C41.7070105@gmail.com> Message-ID: <53DC3D5E.5080909@gmail.com> Sorry, I've forgotten to modified theme according to rules. Sending this message again so anyone can find it more easy. Sorry for being annoying. Hello everyone! Here is my report on progress that was achieved during this time. I've implemented actual Unicode Collation Algorithm for DUCET (Default Unicode Collation Element Table). I had to rewrite the entire implementation: I wasn't satisfied with its quality and the way that I've organized my source code, so I reverted my code and started again. My previous implementation was full of hard-coded parts and it was a bit harder to take anything useful from it for any other project. Now the entire implementation is available in include/unicode.h and lib/libc/unicode. If macro _UNICODE_SOURCE is defined, then wcscoll() will use new collation algorithm. struct _xlocale was modified in the way it will use two new members, colltable and collsize, which are just transmitted to __ucscoll(). If element is not found in the given table or table is NULL, then __ucscoll() tries to find this element in DUCET; if element was not found, then __ucscoll generates collation. I couldn't understand how the alternate shall be used though; it seems that it can be dropped since wcscoll() doesn't has any version that supports tailoring. I left it for now, but I'm pretty sure that we can omit it. I hadn't time to test wcscoll() better (especially using files provided by Unicode Character Database), so this is the task that I will do right now. :-) There are still several ways to improve the speed of the algorithm, but I feel that the time for it hasn't come yet. style(9) issues will also be handled (if any), just too tired to do it right now. __ucscoll() just uses __ucsxfrm(), then compares the strings using wcscmp() (this is the only platform-dependent part of code, I was too lazy to write __ucslen(), so I left it as it is). This collation algorithm support three levels; the last IIRC is usually the character itself if not defined, so I decided to omit it (especially since I'm not sure how variable weights should be handled). Any thoughs? -- With best regards, Dmitry Selyutin From zkorchev at mail.bg Mon Aug 4 09:20:56 2014 From: zkorchev at mail.bg (Zaro Korchev) Date: Mon, 4 Aug 2014 12:20:44 +0300 Subject: [Machine readable output from userland utilities] report In-Reply-To: References: <8D1B686D-1AAA-4E07-9270-E42699110561@mail.bg> <2F779B73-EAC5-49B4-B3E8-7A81109EC166@mail.bg> <5AF0FF20-E41B-4D46-A427-D84D9E2CD047@mail.bg> <19220D3B-101C-45A4-AE41-068E033D5F92@mail.bg> <60B2286C-F0BA-4757-9603-BE23E365A576@mail.bg> Message-ID: <0D20E27A-4281-4B6D-94E3-8707B1AE2E81@mail.bg> Hi everybody This week I worked on libsol. I made some changes to the header file so that applications using libsol do not need to add include path for libsol-required headers. I removed this include path from the Makefiles. I implemented a second format for libsol. It is similar to the ucl format. It is just a proof of concept. At the moment it does not handle special characters in strings properly. The best way to handle special characters will be to use libucl. I found where in its source this functionality is implemented. It is a static function so at the moment I am not able to use it directly. One solution would be to make a global interface to this function. I was looking for libraries with streaming support for XML generation but I have not found such. I plan to add XML support without using an external library. Zaro From nuta at seiya.me Mon Aug 4 15:50:14 2014 From: nuta at seiya.me (Seiya Nuta) Date: Tue, 05 Aug 2014 00:50:04 +0900 Subject: Bootsplash status report #10 Message-ID: <53DFABAC.1020909@seiya.me> Hi all, This is a weekly status report of new enhanced bootsplash. What I had done in this week: * implemented a prompt screen for a passphrase prompt of geom_eli * found some bugs described below What I will do in this week: * fix those bugs Bugs I found: * Screen become insane after bootsplash has finished. * Once I input wrong passphrase, geom_eli does not accept correct passphrase in the second time or after. These bugs are not occurred in the HEAD kernel. I'm investigating the cause of them since last Friday. Reagards, Seiya From tuchalia at gmail.com Mon Aug 4 16:09:21 2014 From: tuchalia at gmail.com (Daniel Peyrolon) Date: Mon, 4 Aug 2014 18:09:18 +0200 Subject: Bootsplash status report #10 In-Reply-To: <53DFABAC.1020909@seiya.me> References: <53DFABAC.1020909@seiya.me> Message-ID: Hello Seiya, I used geom_eli when installed FreeBSD on my laptop, and I can say that I was only able to put my password once. If I failed I just had to restart the machine. What I mean is that that was (at least, during June, or somewhat before) a real bug, happening on HEAD. 2014-08-04 17:50 GMT+02:00 Seiya Nuta : > Hi all, > > This is a weekly status report of new enhanced bootsplash. > > What I had done in this week: > * implemented a prompt screen for a passphrase prompt of geom_eli > * found some bugs described below > > What I will do in this week: > * fix those bugs > > Bugs I found: > * Screen become insane after bootsplash has finished. > * Once I input wrong passphrase, geom_eli does not accept correct > passphrase in the second time or after. > > These bugs are not occurred in the HEAD kernel. > I'm investigating the cause of them since last Friday. > > Reagards, > Seiya > _______________________________________________ > soc-status at freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/soc-status > To unsubscribe, send any mail to "soc-status-unsubscribe at freebsd.org" > -- Daniel From nuta at seiya.me Mon Aug 4 16:27:39 2014 From: nuta at seiya.me (Seiya Nuta) Date: Tue, 05 Aug 2014 01:27:36 +0900 Subject: Bootsplash status report #10 In-Reply-To: References: <53DFABAC.1020909@seiya.me> Message-ID: <53DFB478.7080009@seiya.me> Hello, Thank you for beneficial information. > What I mean is that that was (at least, during June, or somewhat before) a > real bug, happening on HEAD. I created a branch from HEAD in May. That'd explain a lot. I'll merge latest HEAD into my branch. Thank you again for your information! On 8/5/14, 1:09, Daniel Peyrolon wrote: > Hello Seiya, > > I used geom_eli when installed FreeBSD on my laptop, and I can say that I > was only able to put my password once. > If I failed I just had to restart the machine. > > What I mean is that that was (at least, during June, or somewhat before) a > real bug, happening on HEAD. > > > 2014-08-04 17:50 GMT+02:00 Seiya Nuta : > >> Hi all, >> >> This is a weekly status report of new enhanced bootsplash. >> >> What I had done in this week: >> * implemented a prompt screen for a passphrase prompt of geom_eli >> * found some bugs described below >> >> What I will do in this week: >> * fix those bugs >> >> Bugs I found: >> * Screen become insane after bootsplash has finished. >> * Once I input wrong passphrase, geom_eli does not accept correct >> passphrase in the second time or after. >> >> These bugs are not occurred in the HEAD kernel. >> I'm investigating the cause of them since last Friday. >> >> Reagards, >> Seiya >> _______________________________________________ >> soc-status at freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/soc-status >> To unsubscribe, send any mail to "soc-status-unsubscribe at freebsd.org" >> > > > From kczekirda at freebsd.org Mon Aug 4 19:28:29 2014 From: kczekirda at freebsd.org (Kamil Czekirda) Date: Mon, 4 Aug 2014 21:28:26 +0200 Subject: Weekly Status #11 Message-ID: Hi all, In last week I wrote some functionality to "bsdinstall tofile" simulation. Already done: - keymap - hostname - mirror - timezone - services - ipv4 (but need improve) I started ipv6, but not finished. This week I have more free time, so I plan bigger progress. Regards, Kamil Czekirda From bygrandao at gmail.com Tue Aug 5 01:23:15 2014 From: bygrandao at gmail.com (Pedro Arthur) Date: Mon, 4 Aug 2014 22:23:12 -0300 Subject: Weekly report - Lua loader Message-ID: This week I worked on the following tasks: - polish the menu code, added shortcut keys and highlight. - analyze the lua scripts and try to follow a common code style. - ensure that the boot password can't be skipped. These tasks are not complete yet, but I intend to finish them until the next week. From shon.balakrishna at gmail.com Tue Aug 5 01:57:10 2014 From: shon.balakrishna at gmail.com (Shonali Balakrishna) Date: Tue, 5 Aug 2014 07:27:08 +0530 Subject: [BSNMP Enhancements] Status Report # 11 Message-ID: Hi all, This is status report # 11 for the GSoC project - BSNMP Enhancements. Work done in the last week: * Resolved compile errors * Added separate calls for ipv6 and completed pending implementations in bsnmpd + bsnmptools * Modified trap.c & target_snmp.c to support ipv6 Work to be done in the next week: * Work on compile warnings * Test comprehensively The source code changes are mostly located in: https://socsvn.freebsd.org/socsvn/soc2014/shonali/head/contrib/bsnmp/snmpd/ https://socsvn.freebsd.org/socsvn/soc2014/shonali/head/usr.sbin/bsnmpd/tools/ Thanks and Regards, -- Shonali Balakrishna From ghostman.sd at gmail.com Wed Aug 6 01:30:12 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Wed, 6 Aug 2014 05:29:49 +0400 Subject: Report #6: Unicode support Message-ID: Hello everyone! Here are the last news about the Unicode support project[0]. You can always check my repository[1]. During these days I've been working on storing data for Unicode Collation in the more appropriate format than it was before (strange tables with binary search right in C source files). According to one of Pedro's suggestions, I've used types and functions to finish it. I've tried to achive portability for all platforms that support in the way that FreeBSD does (I took care of some subtle things like endianness too). Full set of functions to work with collation databases is provided as well as Python bindings (they were written while creating CLDR database, but seemed to be so useful that I decided to commit them too). Right now code lives under lib/libcolldb directory, though it seems there may be a better place for it (especially for Python bindings). Any suggestions? I'd like to leave this stuff visible (first I wanted to leave it hidden in xlocale_private.h, but I found it really useful) for other developers, but the first what came to my mind was library. I was too tired to rewrite all existing functions to make them support collation databases; I hope to finish it tomorrow. Normalization and canonicalization parts are already done; as it seems, collation itself is also nearing completion, though there is still much to be done. I'd like to thank Pedro and especially Konrad Jankowski, who found the strength to return to his project and gave a helpful hand (and gives right now). There is still much to be done: since I got hooked by this part of work, I couldn't respond to all Pedro's and Konrad's mails during these days. So the nearest targets are to rewrite collation algorithms again to let them work and to begin testing. [0] https://wiki.freebsd.org/SummerOfCode2014/Unicode [1] https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd -- With best regards, Dmitry Selyutin From oliver.pntr at gmail.com Sat Aug 9 15:11:44 2014 From: oliver.pntr at gmail.com (Oliver Pinter) Date: Sat, 9 Aug 2014 17:11:42 +0200 Subject: [intel smap, kpatch] weekly report #10 and #11 Message-ID: Last two week I optimized a little the code and separated machine dependent and independent code. I identified one wrong API usage which i should fix at next week. The KSP framework was mostly done, and tested on real hardware w/ and w/o XSAVEOPT and on qemu w/ and w/o SMAP. The SMAP functionality was tested in qemu with the test framework (what you can find in svn repo or github). The current status is in my wiki. I running again to svn error, when I try to merge current head to my project branch: [...] U sys/dev/usb/usbdevs U sys/dev/hptnr/os_bsd.h U sys/dev/drm2/ttm/ttm_bo_vm.c U sys/dev/acpica/acpi_cpu.c U sys/dev/fdt/simplebus.c U sys/dev/hptrr/hptrr_os_bsd.c U sys/dev/fb/fbd.c svn: E130003: The PROPFIND response contains invalid XML (207 Multi-Status) op at pandora-d ~> svnlite --version svn, version 1.8.8 (r1568071) compiled on bikeshed-malachite-topaz-amber-freebsd What's next: * fix the mentioned bug * more testing * optionally implement variable size patches (not yet required to SMAP functionality) On 7/26/14, Oliver Pinter wrote: > At the last week, I was done with most of the core functionality. The > SMAP capable kernel can boot on w/ and w/o SMAP support. The XSAVEOPT > related manual patching was elliminated and used the common kernel > patchin framework. > > So what's done at this week: > * working kernel patching > * working module patching > * working preload patching > * adopted SMAP instructions to kernel patching > * adopted XSAVEOPT instructions to kernel patching > * tested in Qemu > * tested on real hardware > > What's will I on next week: > * optimize > * fix bugs > * implement other patches than same sized > > On 7/19/14, Oliver Pinter wrote: >> Hi! >> >> This week I mostly implemented the kernel patching framework. It's >> required to optimize a little, but mostly done. The current status can >> you find both in svn or git repo. >> >> The current code boot tested with kernel image patching, it's works. >> Next should I test kld preload patching and kldload patching, and then >> adopting SMAP related instructions and xsave related codes. >> >> Detailed info are in wiki. >> >> On 7/11/14, Oliver Pinter wrote: >>> Hi All! >>> >>> At previous week I started to design a kernel patching framework and I >>> have a little holiday. >>> >>> At this week I mostly finished the design, and started to implement >>> the selfpatching framework. >>> >>> Next week I plan to finish the implementation of the framework, and >>> after that migrate the SMAP stuff to use them. >>> >>> The current status can you found on my wiki site. >>> >>> On 6/28/14, Oliver Pinter wrote: >>>> This week I started the second phase of GSoC. In this design a >>>> run-time kernel and module patching framework. This means that the >>>> kernel able to dynamically change their code run-time. >>>> >>>> In second phase's first week I investigated where must I implement the >>>> functionality and which kernel APIs should I use. >>>> >>>> You can found the current status in my wiki page. >>>> >>>> On 6/21/14, Oliver Pinter wrote: >>>>> Hi! >>>>> >>>>> At this week i am hunting a triple fault during the boot. This caused >>>>> by a compiler error, when CPUTYPE in /etc/make.conf was set to >>>>> core-avx2, after removing this the first phase was done. All of my >>>>> test running fine and the system are stable. Originally only amd64 >>>>> implementation required, but I added to i386 too - but the later not >>>>> yet tested. >>>>> >>>>> In next phase I design a proper way how to patch kernel and modules at >>>>> boot and run-time. >>>>> >>>>> What's done: >>>>> * SMAP for amd64 >>>>> * test SMAP for amd64 >>>>> * build framework >>>>> * VM creation >>>>> * SMAP for i386 (not tested) >>>>> * some other tool, that make my life easier >>>>> >>>>> The current status can you find on my wiki page. >>>>> >>>>> On 6/15/14, Oliver Pinter wrote: >>>>>> Hi all! >>>>>> >>>>>> In the last week I was mostly done with implementation, as you can >>>>>> see >>>>>> on my wiki page. The most of i386 commits are not tested because a >>>>>> cross-build problem on amd64 system. >>>>>> Other resolvable problem are on amd64 system, where the machine >>>>>> triple >>>>>> faulted, because wrong assembler statements generated with the >>>>>> compiler. I'm deep in debugging both of two case. This issue are too >>>>>> in my wiki page under this section: >>>>>> https://wiki.freebsd.org/SummerOfCode2014/IntelSMAPandKernelPatching#notes >>>>>> >>>>>> I have at this week my last exam at Thursday. After that I'm focusing >>>>>> fully on GSoC. >>>>>> >>>>>> On 6/6/14, Oliver Pinter wrote: >>>>>>> Hi all! >>>>>>> >>>>>>> Previous week I started to work on SMAP for amd64 and i386. For >>>>>>> amd64 >>>>>>> many parts are in good state. The codes currently are only compile >>>>>>> tested, at next week I create a VM, and create run-time tests. For >>>>>>> i386 started the work on yesterday. >>>>>>> All of my status can be found on my wiki page. >>>>>>> >>>>>>> What's done, but not tested in this week: >>>>>>> * {amd64,i386} trap handler >>>>>>> * amd64 initialization >>>>>>> * {amd64,i386} identification >>>>>>> * {amd64,i386} exceptions >>>>>>> * amd64 pmap changes >>>>>>> * amd64 support.S changes >>>>>>> * amd64 ia32 compat exceptions >>>>>>> * i386 ddb extension >>>>>>> >>>>>>> At next week I plan to finish all of amd64 things, and most of i386 >>>>>>> things, and begin to test; start to design a proper way to create >>>>>>> kpatch and/or ifunc like things. >>>>>>> >>>>>>> >>>>>>> svn: http://svnweb.freebsd.org/socsvn/soc2014/op/ >>>>>>> git: https://github.com/opntr/opBSD (branches: >>>>>>> op/gsoc2014/{master,smap,kpatch} ) >>>>>>> wiki: >>>>>>> https://wiki.freebsd.org/SummerOfCode2014/IntelSMAPandKernelPatching >>>>>>> >>>>>>> >>>>>>> On 5/29/14, Oliver Pinter wrote: >>>>>>>> Hi all! >>>>>>>> >>>>>>>> I'm working on Intel SMAP technology in first half of GSoC. >>>>>>>> At first week I investigated in SMAP technology and relevant >>>>>>>> FreeBSD >>>>>>>> codes, whats changed since my Bsc thesis. >>>>>>>> >>>>>>>> I implemented a vulnerable kernel module and PoC to test allowed >>>>>>>> and >>>>>>>> not allowed memory access scenario. Created my wiki page, svn repo, >>>>>>>> and git repo. >>>>>>>> >>>>>>>> svn: http://svnweb.freebsd.org/socsvn/soc2014/op/ >>>>>>>> git: https://github.com/opntr/opBSD (branches: >>>>>>>> op/gsoc2014/{master,smap,kpatch} ) >>>>>>>> wiki: >>>>>>>> https://wiki.freebsd.org/SummerOfCode2014/IntelSMAPandKernelPatching >>>>>>>> test-cases: >>>>>>>> http://svnweb.freebsd.org/socsvn/soc2014/op/tests/smap-tester/ >>>>>>>> >>>>>>>> Good days, >>>>>>>> Oliver >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > From zkorchev at mail.bg Mon Aug 11 11:06:42 2014 From: zkorchev at mail.bg (Zaro Korchev) Date: Mon, 11 Aug 2014 14:06:38 +0300 Subject: [Machine readable output from userland utilities] report In-Reply-To: <0D20E27A-4281-4B6D-94E3-8707B1AE2E81@mail.bg> References: <8D1B686D-1AAA-4E07-9270-E42699110561@mail.bg> <2F779B73-EAC5-49B4-B3E8-7A81109EC166@mail.bg> <5AF0FF20-E41B-4D46-A427-D84D9E2CD047@mail.bg> <19220D3B-101C-45A4-AE41-068E033D5F92@mail.bg> <60B2286C-F0BA-4757-9603-BE23E365A576@mail.bg> <0D20E27A-4281-4B6D-94E3-8707B1AE2E81@mail.bg> Message-ID: <176D5B3B-D92A-4E88-B29B-EE109D624ADD@mail.bg> Hi everybody In the last week I worked on libsol. I added xml support to libsol. It is invoked with the appropriate value of the SOL_FORMAT environment variable (xml). I escape the special characters in the string values but I am still not handling them properly in the tag names. I am currently working on supporting special characters in tag names and supporting special characters as a whole in the configuration format. Zaro From nuta at seiya.me Mon Aug 11 11:59:21 2014 From: nuta at seiya.me (Seiya Nuta) Date: Mon, 11 Aug 2014 20:59:10 +0900 Subject: Bootsplash status report #11 Message-ID: <53E8B00E.3070800@seiya.me> Hi all, In last week, I had worked on fixing some bugs and write some documentation, specifically, the wiki page: https://wiki.freebsd.org/SummerOfCode2014/Bootsplash In this week, I plan to refactor code, write documentation and create themes. Regards, Seiya From bygrandao at gmail.com Mon Aug 11 17:46:05 2014 From: bygrandao at gmail.com (Pedro Arthur) Date: Mon, 11 Aug 2014 14:46:03 -0300 Subject: Weecky report - Lua loader Message-ID: This week I worked on the following - Improve lua script code style - Added autoboot timer - Menu to select available kernels I'm now working on the Makefiles to let the user choose which interpreter to use. From kczekirda at freebsd.org Mon Aug 11 20:11:10 2014 From: kczekirda at freebsd.org (Kamil Czekirda) Date: Mon, 11 Aug 2014 22:11:08 +0200 Subject: Weekly Status #11 Message-ID: Hello, First version of fai is ready. It needs some tests, I'll do it this week. How it works? After boot script looks for bootfile-name option in dhcp leases files, dhcp server should tell fai where is directory with configuration files. In next step sctipt looks for file called the same as MAC address from their network interface, tries to download it and it checks for first specific line: "#!fai". If file is wrong script looks for file named "default" and in last step looks for local stored configuration. The last step is runs installer with right configuration file. Few FreeBSD installed ;) Regards, Kamil Czekirda From kczekirda at freebsd.org Mon Aug 11 20:16:27 2014 From: kczekirda at freebsd.org (Kamil Czekirda) Date: Mon, 11 Aug 2014 22:16:24 +0200 Subject: Weekly Status #12 Message-ID: Hello, First version of fai is ready. It needs some tests, I'll do it this week. How it works? After boot script looks for bootfile-name option in dhcp leases files, dhcp server should tell fai where is directory with configuration files. In next step sctipt looks for file called the same as MAC address from their network interface, tries to download it and it checks for first specific line: "#!fai". If file is wrong script looks for file named "default" and in last step looks for local stored configuration. The last step is runs installer with right configuration file. Few FreeBSD installed ;) Regards, Kamil Czekirda From kczekirda at freebsd.org Mon Aug 11 20:16:54 2014 From: kczekirda at freebsd.org (Kamil Czekirda) Date: Mon, 11 Aug 2014 22:16:51 +0200 Subject: Weekly Status #11 In-Reply-To: References: Message-ID: Sorry, no # increment. 2014-08-11 22:11 GMT+02:00 Kamil Czekirda : > Hello, > > First version of fai is ready. It needs some tests, I'll do it this week. > > How it works? > > After boot script looks for bootfile-name option in dhcp leases files, > dhcp server should tell fai where is directory with configuration > files. > > In next step sctipt looks for file called the same as MAC address from > their network interface, tries to download it and it checks for first > specific line: "#!fai". > > If file is wrong script looks for file named "default" and in last > step looks for local stored configuration. > > The last step is runs installer with right configuration file. > > Few FreeBSD installed ;) > > Regards, > Kamil Czekirda From daniel.lovasko at gmail.com Tue Aug 12 15:07:57 2014 From: daniel.lovasko at gmail.com (Daniel Lovasko) Date: Tue, 12 Aug 2014 17:07:56 +0200 Subject: libctf & ddb Message-ID: Hi all, I am _very_ sorry that I forgot to write these weekly reports for few past weeks and I would like to make this right, so here is the work that I have done: I introduced a public API that is created by C macros so that the generic get_something() function is written only once and not N times. All API functions are returning an integer that is one of the CTF_E_* constants or the CTF_OK. If a function is intended to provide some value, it is always transferred by the last argument. All types like ctf_typedef, ctf_enum_entry, ctf_float, ..., are typedefs of appropriate structs and are intended to be used as opaque types. This provides us implementation freedom to the future (we may decide that names are not as char*, but rather some index to a string table or such). All programs that are using this libctf (ctfdump, ctfstats, and ctfquery (see below)) are now using this API. The library is pollution free, as all non-static functions that exist are with the _ctf prefix to ensure the zero-to-none possibility of a name collision. Currently almost every data structure and its member, algorithm and function has its documentation that is written manually. The ctfdump program that serves very similar purpose as the CDDL licensed one. I had a nice idea, that maybe the project "Machine-readable utilities" could provide (with some help, of course) its multi-format support. I have seen some repositories [1] that tried to do this, so there may be an audience for such feature. Also showing the ability to cooperate is very nice indeed. The ctfstats program that computes and emits CTF data statistics (similar to old ctfdump -S). The ctfquery program that serves as an intermediate implementation before putting this code into the DDB. The program takes an input, a type name, and looks it up in the type database. After successful match, it presents the data type in appropriate manner - for typedef it solves the possible typedef chain to the basic type, for struct it prints all its members (and if a member is another struct, it prints it with additional indentation). The only thing missing is the memory access that would make this completely DDB-like. But almost all of this code is usable in the DDB. The type lookup based on the name is using very naive/simple approach - O(n) traversal of all types while reporting success on the first hit. I have done no real benchmarking, since the search appears instantaneous (I even tried simulating some high workload and the cycle run just as swiftly). In case that this is not good enough, there are several possible improvements algorithm-wise - building a trie that would speed the algorithm to O(longest type name) and then making buckets of types - struct bucket, union bucket, enum bucket - so when the user requests e.g. "struct dpt_cbb", we can safely make the enquiry in the struct bucket only. This can be, obviously, applied to all kinds of data structures - simple linked lists or tries. The search works in situations, when the user omits the struct/union/enum part too. One thing that needs separate attention is the ctfquery feature to guess the data structure type - linked list, binary tree, n-ary tree, all the queue(3) and tree(3) data types. The struct in question is tested for presence of all these members (for example, being an queue(3) SLIST means that the struct contains an anonymous struct that has only one member, pointer to the parent-struct and the member has to be named "sle_next". This was maybe the most enjoyable coding from this project so far. The usefulness of this feature is, that after we discover that a struct is a linked list, we are able to print it more intelligently (see my proposal for this). Visualisation of other data structures will not be done in this project, but I am open for future suggestions (but, visualising a red-black tree on a 80x25 terminal might be ... well, challenging). Type to string conversion: if the CTF data in question is for example pointer -> const -> struct dpt_ccb, it gets resolved to "const dpt_ccb*". This is used in the ctfdump and ctfquery programs. There are still some crazy scenarios that need to be taken care of but the majority of the types is converted correctly. The libctf undergone some linting and valgrinding, which discovered some nasty hidden memleaks and potential bugs that are now fixed. One of the bugs took me 4 days to fix - improper handling of large struct members - the mistake was hidden under three layers of logic and I must admit that I was pretty happy after I finally found and fixed it. DDB code to parse arguments of the command (this was a bit tricky thanks to lack of documentation and weird naming). Right now I am fighting a huge problem: while writing the proposal during the last winter, I was able to use linker_ctf_get function to obtain the CTF data of the kernel file in the kernel space. Unfortunately, the same code on the same installation (and on a clean 10.0 and 9.2 installations) crashed very badly and the problem seems to be the vn_open call in the linker_elf_ctf_get in the /usr/sys/kern/kern_ctf.c file. I tried to call the vn_open function directly in my modified DDB code and it crashes too - the exact call looks like this (it is taken directly from the kern_ctf.c file): [2] I am looking forward to any ideas about this problem :) My plans for the next few days: I need to adapt the libctf allocation routines to work in kernel space too, therefore I need to #ifdef the usage of all malloc(3)s with malloc(9)s and some minor changes in strdup, strcpy and such. This should not pose any problem. There is no need for the zlib to be used in the kernel-space-version of the library, as the linker_ctf_get() function returns already unzipped CTF data. Small changes need to be done in the libctf loading code, because right now, we are able to only get file name and read all the ELF sections by ourselves, but the linker_ctf_get() function already does this step, so we can omit this too. To summarize, next baby-step is to be able to print all CTF types inside the DDB and then just copy/paste the ctfquery code and add some usability/user experience functionality like DDB modifiers for hexadecimal output and such. Maybe I forgot some things or details, so if I think of some more additions later, I will write them here. Again, I am sorry for the delay, please do not get an impression of some lazy attitude or that I have not been working on the project. Best, Daniel [1] https://github.com/rmustacc/ctf2json [2] http://pastebin.com/gxG55vHn From ghostman.sd at gmail.com Tue Aug 12 18:05:25 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Tue, 12 Aug 2014 21:47:37 +0400 Subject: Report #7: Unicode support Message-ID: Hello everyone! Here are the last news about the Unicode support project[0]. You can always check my repository[1]. During these days I've been working on integrating changes into the tree. libc now supports UNICODE flag. If it is defined, then the entire libc is compiled with -D_UNICODE_SOURCE, thus supporting Unicode Collation Algorithm as well as Unicode normalization and canonicalization using hidden __ucsnorm() and __ucscanon() functions. Collation Database Library (libcolldb) moved into contrib/, though it has its own Makefile inside lib/ directory. Collation Database Library provides colldb script, which is used to transform Unicode collation files into Collation Database format. There are some things to be done: first I need to create a Makefile that will move contrib/colldb/colldb into /usr/bin (and probably copying it into Python package directory, since this script (if imported) allows to use bindings to libcolldb). This Makefile must also use colldb to create a new database from share/colldb/root.src and install it as /usr/share/colldb/root.db file. Since I'm not sure how to handle such things using BSD make, so I think I'll need your help, Pedro! ;-) Now it can be done manually. The other thing is a more extensive testing using files from Unicode CLDR repository. I've never used FreeBSD testing system, but hopefully it won't be harder than implementing Unicode Collation. [0] https://wiki.freebsd.org/SummerOfCode2014/Unicode [1] https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd -- With best regards, Dmitry Selyutin From shon.balakrishna at gmail.com Wed Aug 13 04:38:40 2014 From: shon.balakrishna at gmail.com (Shonali Balakrishna) Date: Wed, 13 Aug 2014 10:08:37 +0530 Subject: [BSNMP Enhancements] Status Report # 12 Message-ID: Hi all, This is status report # 12 for the GSoC project - BSNMP Enhancements. Work done in the last week: * Reviewed code and added fixes * Added a separate port table for ipv6(begemotSnmpdPortv6Table) in the MIB * Added a separate enum for ipv6 address(SNMP_SYNTAX_IP6ADDRESS) and added calls for the ipv6 functions Work to be done in the next week: * Test comprehensively * Improve code The source code changes are mostly located in: https://socsvn.freebsd.org/socsvn/soc2014/shonali/head/contrib/bsnmp/snmpd/ https://socsvn.freebsd.org/socsvn/soc2014/shonali/head/usr.sbin/bsnmpd/tools/ Thanks and Regards, -- Shonali Balakrishna From jakub.klama at uj.edu.pl Wed Aug 13 23:49:59 2014 From: jakub.klama at uj.edu.pl (Jakub Klama) Date: Thu, 14 Aug 2014 01:49:51 +0200 Subject: evdev status report #11 & #12 Message-ID: <432baa27506a634db7325d3b86edba43@uj.edu.pl> Hello, This is combined status report #11 and #12 for GSoC project named "Implementation of evdev protocol and touchscreens support". There's not much work done as there was not much left to do. Done last two weeks: * fixed linuxishms in libevdev and python-evdev allowing to seamlessly compile them under FreeBSD * made input.h and uinput.h be installed into /usr/include/dev/evdev * written manual page (not yet committed) Next week: * merge changes from HEAD * test everything once again Jakub From oliver.pntr at gmail.com Thu Aug 14 22:01:40 2014 From: oliver.pntr at gmail.com (Oliver Pinter) Date: Thu, 14 Aug 2014 22:48:25 +0200 Subject: [intel smap, kpatch] weekly report #12 - FINAL Message-ID: Last week I finalized the implementation and tested on real hardware and in emulator. All things works fine. You can find all details on wiki page, codes on github and in socsvn, and vm image on my crysys.hu home, and the final patchset in socsvn. Links: https://wiki.freebsd.org/SummerOfCode2014/IntelSMAPandKernelPatching https://github.com/opntr/opBSD/commits/op/gsoc2014/master http://www.crysys.hu/~op/freebsd/vm-image/ http://svnweb.freebsd.org/socsvn/soc2014/op/ On 8/9/14, Oliver Pinter wrote: > Last two week I optimized a little the code and separated machine > dependent and independent code. I identified one wrong API usage which > i should fix at next week. > > The KSP framework was mostly done, and tested on real hardware w/ and > w/o XSAVEOPT and on qemu w/ and w/o SMAP. > The SMAP functionality was tested in qemu with the test framework > (what you can find in svn repo or github). > > The current status is in my wiki. > > I running again to svn error, when I try to merge current head to my > project branch: > > [...] > U sys/dev/usb/usbdevs > U sys/dev/hptnr/os_bsd.h > U sys/dev/drm2/ttm/ttm_bo_vm.c > U sys/dev/acpica/acpi_cpu.c > U sys/dev/fdt/simplebus.c > U sys/dev/hptrr/hptrr_os_bsd.c > U sys/dev/fb/fbd.c > svn: E130003: The PROPFIND response contains invalid XML (207 Multi-Status) > op at pandora-d ~> svnlite --version > svn, version 1.8.8 (r1568071) compiled on > bikeshed-malachite-topaz-amber-freebsd > > What's next: > * fix the mentioned bug > * more testing > * optionally implement variable size patches (not yet required to SMAP > functionality) > > On 7/26/14, Oliver Pinter wrote: >> At the last week, I was done with most of the core functionality. The >> SMAP capable kernel can boot on w/ and w/o SMAP support. The XSAVEOPT >> related manual patching was elliminated and used the common kernel >> patchin framework. >> >> So what's done at this week: >> * working kernel patching >> * working module patching >> * working preload patching >> * adopted SMAP instructions to kernel patching >> * adopted XSAVEOPT instructions to kernel patching >> * tested in Qemu >> * tested on real hardware >> >> What's will I on next week: >> * optimize >> * fix bugs >> * implement other patches than same sized >> >> On 7/19/14, Oliver Pinter wrote: >>> Hi! >>> >>> This week I mostly implemented the kernel patching framework. It's >>> required to optimize a little, but mostly done. The current status can >>> you find both in svn or git repo. >>> >>> The current code boot tested with kernel image patching, it's works. >>> Next should I test kld preload patching and kldload patching, and then >>> adopting SMAP related instructions and xsave related codes. >>> >>> Detailed info are in wiki. >>> >>> On 7/11/14, Oliver Pinter wrote: >>>> Hi All! >>>> >>>> At previous week I started to design a kernel patching framework and I >>>> have a little holiday. >>>> >>>> At this week I mostly finished the design, and started to implement >>>> the selfpatching framework. >>>> >>>> Next week I plan to finish the implementation of the framework, and >>>> after that migrate the SMAP stuff to use them. >>>> >>>> The current status can you found on my wiki site. >>>> >>>> On 6/28/14, Oliver Pinter wrote: >>>>> This week I started the second phase of GSoC. In this design a >>>>> run-time kernel and module patching framework. This means that the >>>>> kernel able to dynamically change their code run-time. >>>>> >>>>> In second phase's first week I investigated where must I implement the >>>>> functionality and which kernel APIs should I use. >>>>> >>>>> You can found the current status in my wiki page. >>>>> >>>>> On 6/21/14, Oliver Pinter wrote: >>>>>> Hi! >>>>>> >>>>>> At this week i am hunting a triple fault during the boot. This caused >>>>>> by a compiler error, when CPUTYPE in /etc/make.conf was set to >>>>>> core-avx2, after removing this the first phase was done. All of my >>>>>> test running fine and the system are stable. Originally only amd64 >>>>>> implementation required, but I added to i386 too - but the later not >>>>>> yet tested. >>>>>> >>>>>> In next phase I design a proper way how to patch kernel and modules >>>>>> at >>>>>> boot and run-time. >>>>>> >>>>>> What's done: >>>>>> * SMAP for amd64 >>>>>> * test SMAP for amd64 >>>>>> * build framework >>>>>> * VM creation >>>>>> * SMAP for i386 (not tested) >>>>>> * some other tool, that make my life easier >>>>>> >>>>>> The current status can you find on my wiki page. >>>>>> >>>>>> On 6/15/14, Oliver Pinter wrote: >>>>>>> Hi all! >>>>>>> >>>>>>> In the last week I was mostly done with implementation, as you can >>>>>>> see >>>>>>> on my wiki page. The most of i386 commits are not tested because a >>>>>>> cross-build problem on amd64 system. >>>>>>> Other resolvable problem are on amd64 system, where the machine >>>>>>> triple >>>>>>> faulted, because wrong assembler statements generated with the >>>>>>> compiler. I'm deep in debugging both of two case. This issue are too >>>>>>> in my wiki page under this section: >>>>>>> https://wiki.freebsd.org/SummerOfCode2014/IntelSMAPandKernelPatching#notes >>>>>>> >>>>>>> I have at this week my last exam at Thursday. After that I'm >>>>>>> focusing >>>>>>> fully on GSoC. >>>>>>> >>>>>>> On 6/6/14, Oliver Pinter wrote: >>>>>>>> Hi all! >>>>>>>> >>>>>>>> Previous week I started to work on SMAP for amd64 and i386. For >>>>>>>> amd64 >>>>>>>> many parts are in good state. The codes currently are only compile >>>>>>>> tested, at next week I create a VM, and create run-time tests. For >>>>>>>> i386 started the work on yesterday. >>>>>>>> All of my status can be found on my wiki page. >>>>>>>> >>>>>>>> What's done, but not tested in this week: >>>>>>>> * {amd64,i386} trap handler >>>>>>>> * amd64 initialization >>>>>>>> * {amd64,i386} identification >>>>>>>> * {amd64,i386} exceptions >>>>>>>> * amd64 pmap changes >>>>>>>> * amd64 support.S changes >>>>>>>> * amd64 ia32 compat exceptions >>>>>>>> * i386 ddb extension >>>>>>>> >>>>>>>> At next week I plan to finish all of amd64 things, and most of i386 >>>>>>>> things, and begin to test; start to design a proper way to create >>>>>>>> kpatch and/or ifunc like things. >>>>>>>> >>>>>>>> >>>>>>>> svn: http://svnweb.freebsd.org/socsvn/soc2014/op/ >>>>>>>> git: https://github.com/opntr/opBSD (branches: >>>>>>>> op/gsoc2014/{master,smap,kpatch} ) >>>>>>>> wiki: >>>>>>>> https://wiki.freebsd.org/SummerOfCode2014/IntelSMAPandKernelPatching >>>>>>>> >>>>>>>> >>>>>>>> On 5/29/14, Oliver Pinter wrote: >>>>>>>>> Hi all! >>>>>>>>> >>>>>>>>> I'm working on Intel SMAP technology in first half of GSoC. >>>>>>>>> At first week I investigated in SMAP technology and relevant >>>>>>>>> FreeBSD >>>>>>>>> codes, whats changed since my Bsc thesis. >>>>>>>>> >>>>>>>>> I implemented a vulnerable kernel module and PoC to test allowed >>>>>>>>> and >>>>>>>>> not allowed memory access scenario. Created my wiki page, svn >>>>>>>>> repo, >>>>>>>>> and git repo. >>>>>>>>> >>>>>>>>> svn: http://svnweb.freebsd.org/socsvn/soc2014/op/ >>>>>>>>> git: https://github.com/opntr/opBSD (branches: >>>>>>>>> op/gsoc2014/{master,smap,kpatch} ) >>>>>>>>> wiki: >>>>>>>>> https://wiki.freebsd.org/SummerOfCode2014/IntelSMAPandKernelPatching >>>>>>>>> test-cases: >>>>>>>>> http://svnweb.freebsd.org/socsvn/soc2014/op/tests/smap-tester/ >>>>>>>>> >>>>>>>>> Good days, >>>>>>>>> Oliver >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > From zkorchev at mail.bg Mon Aug 18 11:01:58 2014 From: zkorchev at mail.bg (Zaro Korchev) Date: Mon, 18 Aug 2014 14:01:53 +0300 Subject: [Machine readable output from userland utilities] report In-Reply-To: <176D5B3B-D92A-4E88-B29B-EE109D624ADD@mail.bg> References: <8D1B686D-1AAA-4E07-9270-E42699110561@mail.bg> <2F779B73-EAC5-49B4-B3E8-7A81109EC166@mail.bg> <5AF0FF20-E41B-4D46-A427-D84D9E2CD047@mail.bg> <19220D3B-101C-45A4-AE41-068E033D5F92@mail.bg> <60B2286C-F0BA-4757-9603-BE23E365A576@mail.bg> <0D20E27A-4281-4B6D-94E3-8707B1AE2E81@mail.bg> <176D5B3B-D92A-4E88-B29B-EE109D624ADD@mail.bg> Message-ID: Hi everybody This week I worked mostly on libsol. I read about XML special characters. I added code that transforms special characters encountered in tag names. This ensures that tag names are valid. I updated the documentation to reflect the changes. I made some changes to the tools using libsol to make sure they don't use characters that will break the XML format. I improved the configuration output format to handle special characters. I also fixed several formatting bugs. The most interesting of them was caused by using a magic pointer value that got optimized by the compiler and lead to wrong output when using the configuration format. Zaro From alexander.tarasikov at gmail.com Mon Aug 18 15:16:38 2014 From: alexander.tarasikov at gmail.com (Alexander Tarasikov) Date: Mon, 18 Aug 2014 19:16:35 +0400 Subject: GSoC final status: porting FreeBSD to Android Emulator Message-ID: Hi FreeBSD/ARM hackers! I am reporting my status on the project. At the start of the project in spring I have written the device drivers for the IRQ chip, timer and framebuffer, but I was unable to verify they work because I needed a working rootfs. I started by using an MD_ROOT ramdisk for debugging init. I have spent most of the summer debugging the random memory corruption which prevented the kernel from booting on Android Emulator. I have tried various methods, like emulating TLS/PCPU with a memory page, but none of them helped. By trying to build a kernel for VERSATILEPB and running in QEMU I have verified it's not the problem in sources/compiler. By porting board code from Android Emulator to QEMU and observing it booting I came to the conclusion that the problem was caused by something in Android Emulator which is based on a very old version of QEMU. Ultimately I decided to try a "nightly" build of emulator, and that has resolved the problem. Unfortunately, I was unable to figure out the precise difference between Linux and FreeBSD MMU management that prevented FreeBSD from booting on regular builds of the emulator, but I have decided to focus on getting device drivers working first. Today I have finished implementing the MMC driver which has allowed to boot into the userland. I have also fixed the timer driver by adding the call to cpu_initclocks_bsp initcall and fixing the timer resolution. I have verified the virtual ethernet driver is also working. I have configured a custom MAC address in qemu and verified the kernel sees it. I have currently pushed the changes against FreeBSD-10 to my git repository. Later this week I plan to clean up the drivers a bit and format the code to look like other FreeBSD driver, rebase it and then push to the GSoC SVN repository. https://github.com/astarasikov/freebsd/commits/android_goldfish_arm_10.0.0?author=astarasikov I used the SD card image for Raspberry Pi which I have obtained from http://ftp4.us.freebsd.org/pub/FreeBSD/snapshots/arm/armv6/ISO-IMAGES/10.0/FreeBSD-10.0-STABLE-arm-armv6-RPI-B-20140729-r269271.img.bz2 Here is the boot log of the first successful boot into userland: http://pastebin.com/ZzReDjDH It is very verbose but in the last lines we can see that it has started init, initialized loopback device and showed and getty prompt. You can obtain the prebuilt kernel image from the following link. I have also updated the wiki page with this information. https://drive.google.com/file/d/0B7wcN-tOkdeRN0lRUDJKa2pWM0U/edit?usp=sharing Please note that the nightly builds of Android Emulator are 64-bit and work on Linux and Mac OS X. To get them working on FreeBSD it would be necessary to compile a 32-bit build. If you are interested in just running Android Emulator on FreeBSD, it works just fine once you install linux_base-f10 or some other linux library set. I will provide the emulator binary later. If anyone wants to make a port, please note that you have to use GCC, because QEMU's TCG is broken when compiled with clang :( Due to wasting a lot of time on MMU issues and debugging the MMC driver, I was unable to write the "events" driver for keyboard input. I plan to do it this week. I will also verify that framebuffer driver is working correctly. I have noticed that a new interface is being developed instead of SYSCONS, so maybe I will migrate the driver to it. My overall impression is positive. Although I have spend time debugging MMU, I have learnt about the UMA memory allocator in the process. I want to resolve the remaining issues in the next two weeks and then I plan to add the support for the OMAP5 System-on-Chip to FreeBSD because I have such board and I want to try out Xen DOMU. -- Regards, Alexander From ghostman.sd at gmail.com Mon Aug 18 18:33:24 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Mon, 18 Aug 2014 22:33:01 +0400 Subject: Report #8: Unicode support Message-ID: Hello everyone! Here are the last news about the Unicode support project[0]. You can always check my repository[1]. During these days I've been testing and fixing a ton of bugs and memory leaks using valgrind and gdb. I've finished libcolldb and its libc internal clone. Each collation database is now a part of reference-counted locale system. If locale is either "C" or "POSIX" or has no collation rules, we use either strcmp() or wcscmp() instead of strcoll() and wcscoll(). If LC_COLLATE file for the given locale not found or is not a collation database, then try to use /usr/share/locale/UTF-8/LC_COLLATE file. If the latter is not found or is not a collation database, we use original algorithm to maintain backward compatibility. In order to generate LC_COLLATE as collation database, one must use share/examples/collation/colldb.py script. usage: colldb.py share/colldb/root.src /usr/share/locale/UTF-8/LC_COLLATE. The C version of this tool is on the way. [0] https://wiki.freebsd.org/SummerOfCode2014/Unicode [1] https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd -- With best regards, Dmitry Selyutin From nuta at seiya.me Tue Aug 19 11:32:00 2014 From: nuta at seiya.me (Seiya Nuta) Date: Tue, 19 Aug 2014 19:36:35 +0900 Subject: Bootsplash status report #12 Message-ID: <53F328B3.9000108@seiya.me> Hi, In the last week, I worked on fixing/writing documentation, refactoring source code and creating a theme. I've managed to finish this project. If you are interested in this, please try it by following the instructions in the wiki: https://wiki.freebsd.org/SummerOfCode2014/Bootsplash Regards, Seiya From shon.balakrishna at gmail.com Wed Aug 20 13:15:20 2014 From: shon.balakrishna at gmail.com (Shonali Balakrishna) Date: Wed, 20 Aug 2014 18:45:16 +0530 Subject: [BSNMP Enhancements] Final Status Report Message-ID: Hi all, This is the final status report for the GSoC project - BSNMP Enhancements. Work done in the last week: * Reviewed code and added improvements * Tested with bsnmpwalk, bsnmpget * Worked on documentation I'm still working on the documentation for this project and will ensure to have it completed and updated in my Wiki by the end of the week. I will also continue contributing to this project after this week and will be testing my code further in the weeks to come. The source code changes are located in: https://socsvn.freebsd.org/socsvn/soc2014/shonali/head/contrib/bsnmp/ https://socsvn.freebsd.org/socsvn/soc2014/shonali/head/usr.sbin/bsnmpd/ Thanks and Regards, -- Shonali Balakrishna From kczekirda at freebsd.org Thu Aug 21 00:02:32 2014 From: kczekirda at freebsd.org (Kamil Czekirda) Date: Thu, 21 Aug 2014 02:02:30 +0200 Subject: Weekly Status #13 - FINAL Message-ID: Hi all, This is final status report of Fully Automatic Installer project. Fai is ready, comments and tests for more scenarios are welcome. Last week I improve some parts of code and wrote part of documentation. I want to do instruction how to use fai step-by-step. Please contact me if you need assistance, not everything is clear on the project wiki page. I would like to thank Devin Teske for comments and comprehensive answers to questions. Regards, Kamil Czekirda From ghostman.sd at gmail.com Tue Aug 26 21:09:21 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Wed, 27 Aug 2014 01:08:58 +0400 Subject: Report #9: Unicode support Message-ID: Hello everyone! Here are the last news about the Unicode support project[0]. You can always check my repository[1]. During these days I had hardware problems (my HDD peacefully died), so development didn't progress so much as before. However, I've eliminated these problems, so I tried to fix bugs and reorganize the code as much as possible. Now everything shall compile. I decided to use __attribute__((constructor)) and __attribute__((destructor)), since I don't know if there exist a better way to open a file once in the startup and closing it when all routines close. I've found one or two occurrences of this construction in FreeBSD code; AFAICT it is rather common in clang and gcc, so I decided to use it. Hopefully it will also allow us to use root collation database in the embedded systems (if any such system really needs collation algorithm). As you may know we need a tool that can convert collation text files obtained from unicode.org to new collation database (colldb) format. There is a version of this tool written in Python (share/examples/colldb/colldb.py). IIRC we can't use Python when we have a base system though, so it seems that we need to written such tool using C language. I was thinking of lex/yacc combo; I've never tried it, but I think it shouldn't be too hard to write a tool using it. I'd like to know your opinions about this task. I've already written a man page (bin/colldb/colldb.1). The only thing which seems dubious is that I decided to use the same name as for the library itself (well, it seems I have a lack of imagination). So we have both colldb.1 and colldb.3 man pages. The other thing I'd really like to do is to really force network byte order in collation database format (I'm sure I've seen a way to do it in Berkley databases). It's a pity that I have no platform with big-endian (or even PDP!) byte order. Any help here is highly appreciated (as well as your thoughts about lex/yacc, i.e. thoughts whether it fits well to my task). Since Google Summer of Code period has passed, I'd like to thank both my mentors, Pedro and David, who gave me a helping hand during this project, and especially Konrad Jankowski, who found time to answer my questions and help me too. Though GSoC is closed, I'd like to stay with FreeBSD project. First of all, I want to finish and bring to mind this project: I don't think it's really finished, especially its testing part, though it seems that new collation algorithm can already be used. Then I'd like to work in other parts of my project, especially in internationalization parts. I'd also like to improve my own library, qc, to provide a rich API for *BSD and POSIX systems, since I acutely feel the lack of such API. If it is possible to stay with project, I'd be very happy to do it. :-) P.S. Does anyone knows how to get diff between only for my branch (i.e. for my part of repository)? svn diff -r $FIRST:$LAST seems to give everything what all FreeBSD's GSoC have done, so I need some other command. Thanks for your help! [0] https://wiki.freebsd.org/SummerOfCode2014/Unicode [1] https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd -- With best regards, Dmitry Selyutin From bapt at FreeBSD.org Tue Aug 26 22:16:16 2014 From: bapt at FreeBSD.org (Baptiste Daroussin) Date: Wed, 27 Aug 2014 00:16:10 +0200 Subject: Report #9: Unicode support In-Reply-To: References: Message-ID: <20140826221610.GD65120@ivaldir.etoilebsd.net> On Wed, Aug 27, 2014 at 01:08:58AM +0400, Dmitry Selyutin wrote: > Hello everyone! > > Here are the last news about the Unicode support project[0]. > You can always check my repository[1]. > > During these days I had hardware problems (my HDD peacefully died), so > development didn't progress so much as before. However, I've > eliminated these problems, so I tried to fix bugs and reorganize the > code as much as possible. Now everything shall compile. > > I decided to use __attribute__((constructor)) and > __attribute__((destructor)), since I don't know if there exist a > better way to open a file once in the startup and closing it when all > routines close. I've found one or two occurrences of this construction > in FreeBSD code; AFAICT it is rather common in clang and gcc, so I > decided to use it. Hopefully it will also allow us to use root > collation database in the embedded systems (if any such system really > needs collation algorithm). > > As you may know we need a tool that can convert collation text files > obtained from unicode.org to new collation database (colldb) format. > There is a version of this tool written in Python > (share/examples/colldb/colldb.py). IIRC we can't use Python when we > have a base system though, so it seems that we need to written such > tool using C language. I was thinking of lex/yacc combo; I've never > tried it, but I think it shouldn't be too hard to write a tool using > it. I'd like to know your opinions about this task. > I've already written a man page (bin/colldb/colldb.1). The only thing > which seems dubious is that I decided to use the same name as for the > library itself (well, it seems I have a lack of imagination). So we > have both colldb.1 and colldb.3 man pages. > > The other thing I'd really like to do is to really force network byte > order in collation database format (I'm sure I've seen a way to do it > in Berkley databases). It's a pity that I have no platform with > big-endian (or even PDP!) byte order. Any help here is highly > appreciated (as well as your thoughts about lex/yacc, i.e. thoughts > whether it fits well to my task). > > Since Google Summer of Code period has passed, I'd like to thank both > my mentors, Pedro and David, who gave me a helping hand during this > project, and especially Konrad Jankowski, who found time to answer my > questions and help me too. Though GSoC is closed, I'd like to stay > with FreeBSD project. First of all, I want to finish and bring to mind > this project: I don't think it's really finished, especially its > testing part, though it seems that new collation algorithm can already > be used. Then I'd like to work in other parts of my project, > especially in internationalization parts. I'd also like to improve my > own library, qc, to provide a rich API for *BSD and POSIX systems, > since I acutely feel the lack of such API. If it is possible to stay > with project, I'd be very happy to do it. :-) > > P.S. Does anyone knows how to get diff between only for my branch > (i.e. for my part of repository)? svn diff -r $FIRST:$LAST seems to > give everything what all FreeBSD's GSoC have done, so I need some > other command. Thanks for your help! > > [0] https://wiki.freebsd.org/SummerOfCode2014/Unicode > [1] https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd > First thank you very much for your work on this subject this is highly needed. Concerning the db format have you thought about using the new netbsd constant database format? It has simple API way easier to use, the db format is endian safe and final file is smaller than equivalent in bdb format. Lots of areas of FreeBSD could benefit from using this cdb format as well imho. regards, Bapt -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 181 bytes Desc: not available URL: From pfg at freebsd.org Tue Aug 26 23:17:40 2014 From: pfg at freebsd.org (Pedro Giffuni) Date: Tue, 26 Aug 2014 18:17:45 -0500 Subject: Report #9: Unicode support In-Reply-To: <20140826221610.GD65120@ivaldir.etoilebsd.net> References: <20140826221610.GD65120@ivaldir.etoilebsd.net> Message-ID: <53FD1599.7040708@freebsd.org> Hi Baptiste; On 08/26/14 17:16, Baptiste Daroussin wrote: > On Wed, Aug 27, 2014 at 01:08:58AM +0400, Dmitry Selyutin wrote: >> Hello everyone! >> >> Here are the last news about the Unicode support project[0]. >> You can always check my repository[1]. >> >> During these days I had hardware problems (my HDD peacefully died), so >> development didn't progress so much as before. However, I've >> eliminated these problems, so I tried to fix bugs and reorganize the >> code as much as possible. Now everything shall compile. >> >> I decided to use __attribute__((constructor)) and >> __attribute__((destructor)), since I don't know if there exist a >> better way to open a file once in the startup and closing it when all >> routines close. I've found one or two occurrences of this construction >> in FreeBSD code; AFAICT it is rather common in clang and gcc, so I >> decided to use it. Hopefully it will also allow us to use root >> collation database in the embedded systems (if any such system really >> needs collation algorithm). >> >> As you may know we need a tool that can convert collation text files >> obtained from unicode.org to new collation database (colldb) format. >> There is a version of this tool written in Python >> (share/examples/colldb/colldb.py). IIRC we can't use Python when we >> have a base system though, so it seems that we need to written such >> tool using C language. I was thinking of lex/yacc combo; I've never >> tried it, but I think it shouldn't be too hard to write a tool using >> it. I'd like to know your opinions about this task. >> I've already written a man page (bin/colldb/colldb.1). The only thing >> which seems dubious is that I decided to use the same name as for the >> library itself (well, it seems I have a lack of imagination). So we >> have both colldb.1 and colldb.3 man pages. >> >> The other thing I'd really like to do is to really force network byte >> order in collation database format (I'm sure I've seen a way to do it >> in Berkley databases). It's a pity that I have no platform with >> big-endian (or even PDP!) byte order. Any help here is highly >> appreciated (as well as your thoughts about lex/yacc, i.e. thoughts >> whether it fits well to my task). >> >> Since Google Summer of Code period has passed, I'd like to thank both >> my mentors, Pedro and David, who gave me a helping hand during this >> project, and especially Konrad Jankowski, who found time to answer my >> questions and help me too. Though GSoC is closed, I'd like to stay >> with FreeBSD project. First of all, I want to finish and bring to mind >> this project: I don't think it's really finished, especially its >> testing part, though it seems that new collation algorithm can already >> be used. Then I'd like to work in other parts of my project, >> especially in internationalization parts. I'd also like to improve my >> own library, qc, to provide a rich API for *BSD and POSIX systems, >> since I acutely feel the lack of such API. If it is possible to stay >> with project, I'd be very happy to do it. :-) >> >> P.S. Does anyone knows how to get diff between only for my branch >> (i.e. for my part of repository)? svn diff -r $FIRST:$LAST seems to >> give everything what all FreeBSD's GSoC have done, so I need some >> other command. Thanks for your help! >> >> [0] https://wiki.freebsd.org/SummerOfCode2014/Unicode >> [1] https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd >> > First thank you very much for your work on this subject this is highly needed. > > Concerning the db format have you thought about using the new netbsd constant > database format? > > It has simple API way easier to use, the db format is endian safe and final file > is smaller than equivalent in bdb format. > > Lots of areas of FreeBSD could benefit from using this cdb format as well imho. While here, let me congratulate Dmitry. The Unicode Collation Algorithm is not something easy/fun to work with. Indeed both David and Konrad suggested it (or tinycdb). The reason for going bdb was that we had time constraints and bdb is already in libc. FWIW, Nexenta kindly re-licensed localedef [1] and their collation support in Illumos which basically implements their own very efficient format. We ended up re-using the tools that libc already has to better focus on the collation part. Changing it to use the NetBSD's cdb support[1] shouldn't be difficult. As Dmitry noted there are still details to work out and we have to run tests and get the code reviewed but all in all I am very satisfied with the advance in this GSoC. Best regards, Pedro. [1] https://github.com/Nexenta/illumos-nexenta/tree/republish-localedef [2] http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/cdb/ From ghostman.sd at gmail.com Wed Aug 27 10:48:34 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Wed, 27 Aug 2014 14:48:09 +0400 Subject: Report #9: Unicode support In-Reply-To: <53FD1599.7040708@freebsd.org> References: <20140826221610.GD65120@ivaldir.etoilebsd.net> <53FD1599.7040708@freebsd.org> Message-ID: Hi, Pedro, Baptiste, first of all thanks for your congratulations and kind words! The project was really harder that anything I've ever met in my life, but at the same time it was the most interesting one. :-) And still remains! ;-) > That is not really uncommon :) Well, so I can leave it as it is. :-) > The project does have access to sparc64 machines so if you have some > self-contained test we can run it for you or we can test it as a routine libc > test after committing. Hopefully I can finish it today or in the next two days. > You never answered my question concerning the fallback options. Really? I thought that I answered. :-D Well, I'll try to explain again. DUCET seems to be a bit obsolete collation table, which can be more or less successfully used with real languages. However, in real world it is completely unusable, so ICU and other use CLDR collation table, which supports more levels. I started with DUCET since there was much more information about it, but then I found that it doesn't fit well, so I switched to CLDR. We have DUCET table somewhere in our revisions though; as a fallback option, it still may be useful, so I can restore it if you want. > Changing it to use the NetBSD's cdb support[1] shouldn't be difficult. Well, I think I'll do it right after exams. bdb AFAIK is deprecated from Linux (though it can be used as bdb46 or something similar). I don't know reasons why they did such thing; it would be great if we could use a tool which can be used on different platforms without modifications and tons of conditional define's and undef's. > It has simple API way easier to use, the db format is endian safe and final file > is smaller than equivalent in bdb format. It sounds great! > I do want to encourage you to go to EuroBSDCon 2014 in Sofia. The > FreeBSD Foundation will be allocating funds for students that want to go. > I won?t be there (I am a bit far away) but David and other developers will > likely be. Well, that depends on whether I pass my exams for the postgraduate course or not. I'd really like to listen to more experienced developers and may be even talk to other people about work which I did to better understand the community's opinions. 2014-08-27 3:17 GMT+04:00 Pedro Giffuni : > Hi Baptiste; > > > On 08/26/14 17:16, Baptiste Daroussin wrote: >> >> On Wed, Aug 27, 2014 at 01:08:58AM +0400, Dmitry Selyutin wrote: >>> >>> Hello everyone! >>> >>> Here are the last news about the Unicode support project[0]. >>> You can always check my repository[1]. >>> >>> During these days I had hardware problems (my HDD peacefully died), so >>> development didn't progress so much as before. However, I've >>> eliminated these problems, so I tried to fix bugs and reorganize the >>> code as much as possible. Now everything shall compile. >>> >>> I decided to use __attribute__((constructor)) and >>> __attribute__((destructor)), since I don't know if there exist a >>> better way to open a file once in the startup and closing it when all >>> routines close. I've found one or two occurrences of this construction >>> in FreeBSD code; AFAICT it is rather common in clang and gcc, so I >>> decided to use it. Hopefully it will also allow us to use root >>> collation database in the embedded systems (if any such system really >>> needs collation algorithm). >>> >>> As you may know we need a tool that can convert collation text files >>> obtained from unicode.org to new collation database (colldb) format. >>> There is a version of this tool written in Python >>> (share/examples/colldb/colldb.py). IIRC we can't use Python when we >>> have a base system though, so it seems that we need to written such >>> tool using C language. I was thinking of lex/yacc combo; I've never >>> tried it, but I think it shouldn't be too hard to write a tool using >>> it. I'd like to know your opinions about this task. >>> I've already written a man page (bin/colldb/colldb.1). The only thing >>> which seems dubious is that I decided to use the same name as for the >>> library itself (well, it seems I have a lack of imagination). So we >>> have both colldb.1 and colldb.3 man pages. >>> >>> The other thing I'd really like to do is to really force network byte >>> order in collation database format (I'm sure I've seen a way to do it >>> in Berkley databases). It's a pity that I have no platform with >>> big-endian (or even PDP!) byte order. Any help here is highly >>> appreciated (as well as your thoughts about lex/yacc, i.e. thoughts >>> whether it fits well to my task). >>> >>> Since Google Summer of Code period has passed, I'd like to thank both >>> my mentors, Pedro and David, who gave me a helping hand during this >>> project, and especially Konrad Jankowski, who found time to answer my >>> questions and help me too. Though GSoC is closed, I'd like to stay >>> with FreeBSD project. First of all, I want to finish and bring to mind >>> this project: I don't think it's really finished, especially its >>> testing part, though it seems that new collation algorithm can already >>> be used. Then I'd like to work in other parts of my project, >>> especially in internationalization parts. I'd also like to improve my >>> own library, qc, to provide a rich API for *BSD and POSIX systems, >>> since I acutely feel the lack of such API. If it is possible to stay >>> with project, I'd be very happy to do it. :-) >>> >>> P.S. Does anyone knows how to get diff between only for my branch >>> (i.e. for my part of repository)? svn diff -r $FIRST:$LAST seems to >>> give everything what all FreeBSD's GSoC have done, so I need some >>> other command. Thanks for your help! >>> >>> [0] https://wiki.freebsd.org/SummerOfCode2014/Unicode >>> [1] https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd >>> >> First thank you very much for your work on this subject this is highly >> needed. >> >> Concerning the db format have you thought about using the new netbsd >> constant >> database format? >> >> It has simple API way easier to use, the db format is endian safe and >> final file >> is smaller than equivalent in bdb format. >> >> Lots of areas of FreeBSD could benefit from using this cdb format as well >> imho. > > > While here, let me congratulate Dmitry. The Unicode Collation Algorithm is > not something easy/fun to work with. > > Indeed both David and Konrad suggested it (or tinycdb). The reason for > going bdb was that we had time constraints and bdb is already in libc. > > FWIW, Nexenta kindly re-licensed localedef [1] and their collation support > in Illumos which basically implements their own very efficient format. We > ended up re-using the tools that libc already has to better focus on the > collation part. > > Changing it to use the NetBSD's cdb support[1] shouldn't be difficult. > > As Dmitry noted there are still details to work out and we have to run tests > and get the code reviewed but all in all I am very satisfied with the > advance > in this GSoC. > > Best regards, > > Pedro. > > [1] https://github.com/Nexenta/illumos-nexenta/tree/republish-localedef > [2] http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/cdb/ > -- With best regards, Dmitry Selyutin From ghostman.sd at gmail.com Wed Aug 27 10:51:24 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Wed, 27 Aug 2014 14:51:01 +0400 Subject: Report #9: Unicode support In-Reply-To: References: <20140826221610.GD65120@ivaldir.etoilebsd.net> <53FD1599.7040708@freebsd.org> Message-ID: I've just seen EuroBSDCon's calendar page and it seems that it is impossible to join it (i.e. I missed the application deadline).[0] Well, may be next year? :-) 2014-08-27 14:48 GMT+04:00 Dmitry Selyutin : > Hi, Pedro, Baptiste, > > first of all thanks for your congratulations and kind words! The > project was really harder that anything I've ever met in my life, but > at the same time it was the most interesting one. :-) And still > remains! ;-) > >> That is not really uncommon :) > Well, so I can leave it as it is. :-) > >> The project does have access to sparc64 machines so if you have some >> self-contained test we can run it for you or we can test it as a routine libc >> test after committing. > Hopefully I can finish it today or in the next two days. > >> You never answered my question concerning the fallback options. > Really? I thought that I answered. :-D Well, I'll try to explain > again. DUCET seems to be a bit obsolete collation table, which can be > more or less successfully used with real languages. However, in real > world it is completely unusable, so ICU and other use CLDR collation > table, which supports more levels. I started with DUCET since there > was much more information about it, but then I found that it doesn't > fit well, so I switched to CLDR. We have DUCET table somewhere in our > revisions though; as a fallback option, it still may be useful, so I > can restore it if you want. > >> Changing it to use the NetBSD's cdb support[1] shouldn't be difficult. > Well, I think I'll do it right after exams. bdb AFAIK is deprecated > from Linux (though it can be used as bdb46 or something similar). I > don't know reasons why they did such thing; it would be great if we > could use a tool which can be used on different platforms without > modifications and tons of conditional define's and undef's. > >> It has simple API way easier to use, the db format is endian safe and final file >> is smaller than equivalent in bdb format. > It sounds great! > >> I do want to encourage you to go to EuroBSDCon 2014 in Sofia. The >> FreeBSD Foundation will be allocating funds for students that want to go. >> I won?t be there (I am a bit far away) but David and other developers will >> likely be. > Well, that depends on whether I pass my exams for the postgraduate > course or not. I'd really like to listen to more experienced > developers and may be even talk to other people about work which I did > to better understand the community's opinions. > > 2014-08-27 3:17 GMT+04:00 Pedro Giffuni : >> Hi Baptiste; >> >> >> On 08/26/14 17:16, Baptiste Daroussin wrote: >>> >>> On Wed, Aug 27, 2014 at 01:08:58AM +0400, Dmitry Selyutin wrote: >>>> >>>> Hello everyone! >>>> >>>> Here are the last news about the Unicode support project[0]. >>>> You can always check my repository[1]. >>>> >>>> During these days I had hardware problems (my HDD peacefully died), so >>>> development didn't progress so much as before. However, I've >>>> eliminated these problems, so I tried to fix bugs and reorganize the >>>> code as much as possible. Now everything shall compile. >>>> >>>> I decided to use __attribute__((constructor)) and >>>> __attribute__((destructor)), since I don't know if there exist a >>>> better way to open a file once in the startup and closing it when all >>>> routines close. I've found one or two occurrences of this construction >>>> in FreeBSD code; AFAICT it is rather common in clang and gcc, so I >>>> decided to use it. Hopefully it will also allow us to use root >>>> collation database in the embedded systems (if any such system really >>>> needs collation algorithm). >>>> >>>> As you may know we need a tool that can convert collation text files >>>> obtained from unicode.org to new collation database (colldb) format. >>>> There is a version of this tool written in Python >>>> (share/examples/colldb/colldb.py). IIRC we can't use Python when we >>>> have a base system though, so it seems that we need to written such >>>> tool using C language. I was thinking of lex/yacc combo; I've never >>>> tried it, but I think it shouldn't be too hard to write a tool using >>>> it. I'd like to know your opinions about this task. >>>> I've already written a man page (bin/colldb/colldb.1). The only thing >>>> which seems dubious is that I decided to use the same name as for the >>>> library itself (well, it seems I have a lack of imagination). So we >>>> have both colldb.1 and colldb.3 man pages. >>>> >>>> The other thing I'd really like to do is to really force network byte >>>> order in collation database format (I'm sure I've seen a way to do it >>>> in Berkley databases). It's a pity that I have no platform with >>>> big-endian (or even PDP!) byte order. Any help here is highly >>>> appreciated (as well as your thoughts about lex/yacc, i.e. thoughts >>>> whether it fits well to my task). >>>> >>>> Since Google Summer of Code period has passed, I'd like to thank both >>>> my mentors, Pedro and David, who gave me a helping hand during this >>>> project, and especially Konrad Jankowski, who found time to answer my >>>> questions and help me too. Though GSoC is closed, I'd like to stay >>>> with FreeBSD project. First of all, I want to finish and bring to mind >>>> this project: I don't think it's really finished, especially its >>>> testing part, though it seems that new collation algorithm can already >>>> be used. Then I'd like to work in other parts of my project, >>>> especially in internationalization parts. I'd also like to improve my >>>> own library, qc, to provide a rich API for *BSD and POSIX systems, >>>> since I acutely feel the lack of such API. If it is possible to stay >>>> with project, I'd be very happy to do it. :-) >>>> >>>> P.S. Does anyone knows how to get diff between only for my branch >>>> (i.e. for my part of repository)? svn diff -r $FIRST:$LAST seems to >>>> give everything what all FreeBSD's GSoC have done, so I need some >>>> other command. Thanks for your help! >>>> >>>> [0] https://wiki.freebsd.org/SummerOfCode2014/Unicode >>>> [1] https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd >>>> >>> First thank you very much for your work on this subject this is highly >>> needed. >>> >>> Concerning the db format have you thought about using the new netbsd >>> constant >>> database format? >>> >>> It has simple API way easier to use, the db format is endian safe and >>> final file >>> is smaller than equivalent in bdb format. >>> >>> Lots of areas of FreeBSD could benefit from using this cdb format as well >>> imho. >> >> >> While here, let me congratulate Dmitry. The Unicode Collation Algorithm is >> not something easy/fun to work with. >> >> Indeed both David and Konrad suggested it (or tinycdb). The reason for >> going bdb was that we had time constraints and bdb is already in libc. >> >> FWIW, Nexenta kindly re-licensed localedef [1] and their collation support >> in Illumos which basically implements their own very efficient format. We >> ended up re-using the tools that libc already has to better focus on the >> collation part. >> >> Changing it to use the NetBSD's cdb support[1] shouldn't be difficult. >> >> As Dmitry noted there are still details to work out and we have to run tests >> and get the code reviewed but all in all I am very satisfied with the >> advance >> in this GSoC. >> >> Best regards, >> >> Pedro. >> >> [1] https://github.com/Nexenta/illumos-nexenta/tree/republish-localedef >> [2] http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/cdb/ >> > > > > -- > With best regards, > Dmitry Selyutin -- With best regards, Dmitry Selyutin From pfg at freebsd.org Wed Aug 27 15:28:02 2014 From: pfg at freebsd.org (Pedro Giffuni) Date: Wed, 27 Aug 2014 10:28:11 -0500 Subject: Report #9: Unicode support In-Reply-To: References: <20140826221610.GD65120@ivaldir.etoilebsd.net> <53FD1599.7040708@freebsd.org> Message-ID: <53FDF90B.4030400@freebsd.org> On 08/27/14 05:51, Dmitry Selyutin wrote: > ... >>> You never answered my question concerning the fallback options. >> Really? I thought that I answered. :-D Well, I'll try to explain >> again. DUCET seems to be a bit obsolete collation table, which can be >> more or less successfully used with real languages. However, in real >> world it is completely unusable, so ICU and other use CLDR collation >> table, which supports more levels. I started with DUCET since there >> was much more information about it, but then I found that it doesn't >> fit well, so I switched to CLDR. We have DUCET table somewhere in our >> revisions though; as a fallback option, it still may be useful, so I >> can restore it if you want. I don't see DUCET as being ever used but we are setting the old algorithm as a fallback for CLDR. I was just wondering how DUCET compares to the existing algorithm. Given that DUCET is in the standard and that you already implemented it, I thought it would be a better fallback than the old code. It's your call though. Pedro. From ghostman.sd at gmail.com Thu Aug 28 20:23:41 2014 From: ghostman.sd at gmail.com (Dmitry Selyutin) Date: Fri, 29 Aug 2014 00:23:18 +0400 Subject: Report #10: Unicode support Message-ID: Hello everyone! I've written a colldb tool using C language, so one may now use colldb after compiling sources. I decided not to use lex/yacc here since it seemed to be a bit overkill for just a simple task, so all you need is C plus libcolldb (written in C too). I've also written colldb.1 manual page and fixed libcolldb build (previously one could not compile it without UNICODE=YES in make.conf). Due to my exams (they will start in several days) I have to take a rest to be prepared. Thanks to everyone who helped me during this summer! I'd like to thank FreeBSD's community: guys, you are amazing, it's really pleasant to work with you! I'm going to continue my work after September 20th, so stay tuned! ;-) P.S. Repository is here as usual: https://socsvn.freebsd.org/socsvn/soc2014/ghostmansd -- With best regards, Dmitry Selyutin From bygrandao at gmail.com Sat Aug 30 19:13:33 2014 From: bygrandao at gmail.com (Pedro Arthur) Date: Sat, 30 Aug 2014 16:13:30 -0300 Subject: Lua loader Message-ID: Hi, I updated my wiki page (#New_milestones ) with the proposed fixes needed to integrate the work. I intend to work on these fixes in the coming weeks. I also would like to thanks my mentor Wojciech and all the FreeBSD community, for accepting my project and support me. It was a great experience and I hope to have a chance to work together with you in the future.