BSDL texttools status and further thoughts...

Gabor Kovesdan gabor at
Sun Sep 27 13:27:09 UTC 2009

Hello all,

recently, I've had a discussion with rdivacky@ about the status of these 
tools. It's about bc, dc, grep, sort and iconv. He has persuaded me to 
write a summary here in case someone else is interested in contributing 
to these tools. So here I come with a little summary.

BSD bc/dc will come just after 8.0-RELEASE. They are quite mature and 
delphij@ offered to help me getting this into the three by reviewing and 
approving my changes (I only have doc/ports bit).

BSD grep is also quite mature, I've fixed the last critical bug 
recently. My only concern is the performance. GNU is fast but has ~8 
KSLOC. BSD grep is slightly slower but has only ~1.5 KSLOC. It's a huge 
difference in complexity and GNU grep is very hard to read but they use 
a lot of custom optimizations to get this performance. I think we should 
go another way and have a well-optimized and mature regex library. The 
current one is very old and doesn't have wchar support, it's slow like 
hell and doesn't support custom GNU bullshit, which is unfortunately 
necessary to maintain compatiblity. (e.g. "(a|)" is considered invalid 
in strict POSIX regex but GNU accepts it!) Because of this, BSD grep is 
linked to the GNU regex library at the moment but because of the custom 
magic in grep it's still slower a bit. If we can live with this slight 
performance hit, we can commit it, I think because it's quite 
feature-complete. You know, I'm a beginner but I think that the code of 
BSD grep is so tiny and simple that there are almost absolutely no ways 
to optimize it more by simplifying the code, so I think further 
optimization should be done in the regex library. As for the regex 
library, NetBSD's SoC project is worth a look. I'm interested in this 
but I have too much things in the queue to start another one...

As for sort, it isn't so mature yet. I've just made a TODO list of the 
known missing features or bugs:
- sometimes it segfaults when reading huge files
- the -k option isn't implemented yet
- the -n option doesn't work correctly
- preproc() optimization (I don't what it refers to actually but I had 
it on my previous TODO list, will have to check)
- polishing man page
- adding some more test cases to the regression test
- checking performance (in this case, it really matters because sorting 
is an algorithmic piece of cake and I'm not an algorithmic guru... And 
this version of sort was written by me from scratch. The OpenBSD-one 
isn't wchar-clean and can't be fixed by design.  This sort is much more 
tiny but it seems the algorithm isn't optimal.)

As for iconv, I'll keep working on it in my BSc thesis. The forward (foo 
-> utf32) conversions are almost completely GNU-compatible, the reverse 
ones not so much. GNU has an optional transliteration, while BSD iconv 
uses it at default so I compared the output to GNU's transliterated 
output and it has some more advanced mappings to do this. Apart from 
this, almost all encodings are supported, that we have in locale(1) 
charmaps but the Big5 module segfaults. I hope I'll be able to solve 
these issues and check performance as part of my BSc thesis.


Gabor Kovesdan
FreeBSD Volunteer

EMAIL: gabor at .:|:. gabor at
WEB: .:|:.

More information about the freebsd-hackers mailing list