GSoC proposal: Quirinus C library (qc)

Wed Feb 26 21:01:45 UTC 2014

Hi Dmitry,

Let me first just say that this is a great idea - libc has always traditionally been a very minimalist place to start writing applications from, and certainly “lack of high-level API” is one of the traditional Achilles heels of Unix, not just FreeBSD.  I can certainly say from personal experience that the life of an ISV on a Unix platform is a sad one - you either end up writing a lot of the foundational libraries yourself and/or you stick together a potpourri of libraries like apr, glib and gmp to try and give your application a somewhat higher place to stand, hoping like hell of course that the foundational libraries you’ve chosen actually work, and work together!

That said, I feel compelled to point out that you’re also about to drive into a swamp, and there are lots of alligators and poisonous snakes in there. :-)

If you’re going to do this, it’s best to go into it with your eyes fully open and aware of the magnitude of the challenge you’re about to undertake (hint - you will be at least 28 years old before you consider this “done” enough to walk away, or more likely run screaming, from it because by then you’ll be really really ready to move onto something else!).   I don’t say this to discourage you, I just want to make sure you’re fully aware of what you’re about to undertake if you’re truly committed to seeing it all the way through - this is far more than just a GSoC project!

One of the big reasons it’s such a swamp is the degree to which things quickly become interconnected inside such a library.

You want everything to be I18N compliant by default, so first you implement the Unicode character handling support and the relevant character classing functions, and of course Unicode also means strings, so you implement your own unicode-aware String type and also add a bunch of convenience functions to Strings while you’re in there.  Then, naturally, you need to convert other data types (like C strings) to and from your String type, so you add those functions (and if you want to see how far that can go - look here!).  Once you’ve done all that work, you want the rest of your library to use your String type, so now you’ve got your second (after the character handling) bit of inter-connectedness and it just multiplies from there.   All your base types (Strings, Hashes, Arrays, Trees, etc) become consumed by the rest of the library and any client of gc_lib soon needs to sign up for the whole package - they can’t just pick and choose individual functions as they see fit, and maybe that’s OK, but that’s part of the reason why there are so many competing libraries like this - everyone wants to own the foundation because it’s just so much easier when they do!

Perhaps all of this has already occurred to you, in which case I’m just stating the obvious, but having written such libraries myself (libxtr - stood for “extra functions” - back in the 1990’s) and also having had recent experience with Apple’s Foundation libraries, which do everything you just described and a lot more, I figured it wouldn’t hurt to point it out.  Anyway, on to the specifics:

On Feb 25, 2014, at 12:24 AM, Dmitry Selyutin <ghostman.sd at gmail.com> wrote:

> 1. Strict and universal interface. Each function begins with qc_ prefix,
> followed by type if function is type-related.

Yep, it’s good you put that first.  If the names don’t make sense, nobody can remember what to call and the library is useless!  Uniform calling conventions for everything is also obviously important.

> 4. Universal file system path type. It is possible to achieve
> cross-platformness using such path types, i.e. it is possible to make
> directories, links, symlinks, remove files, directories, etc. regardless of
> platform path API.

Here, however, I think you might want to really consider the role of this library.   Is it a “higher level library for BSD” or is it a cross-platform development library that caters to non-Unix platforms as well?  You mentioned Windows earlier in the document, and if so then that’s cause for concern because if you really want to make it a cross-platform library then you’re going to be faced with lots of different constraints that just don’t apply to FreeBSD and will seriously limit the approaches you can take.  Moreover, the code itself will be a lot uglier if it has to compile on Windows (or $someOtherNonUnixOS) as well, and you will spend a substantial amount of time and energy just impedance matching the library to the different OS foundations.

Personally, I think it should be a non-goal to make this library cross-platform.  There are already plenty of other good cross-platform development libraries out there if that’s what an ISV wants to do, whereas there is no “better libc than libc” library for *BSD, which holds native development back.  More importantly, if you just focus on BSD as your foundation, you can make a lot of assumptions and leverage the power of the underlying OS in ways you just won’t be able to do if you want this to also work on Windows.

> 6. Multithreading support if platform permits it. On POSIX we use pthreads,
> on Windows we use its API for multithreading. I'm also thinking about green
> threads implementation. Thread local storage is already implemented, yet
> there is still a lot work to be done with threads, mutexes, etc.

Please don’t reinvent pthreads.  They already suck enough in libc. :-)

I’ve been up on the libdispatch soapbox so many times that I’m sure people here are tired of hearing it, but seriously, once Grand Central Dispatch became firmly embedded in iOS and OS X, the app writers never looked back at pthreads again because GCD is just so much easier to use, and takes so much of the pain out of parallelism, that I’ve received countless emails from developers crying that they were recently forced to go back to pthreads on some other platform and had forgotten how horrible and painful the experience was!   GCD has also been ported to Windows, Linux and Android (again, by some of those same developers who just would not, could not, go back to pthreads on Android) largely on the strength of how much better an API it is for multi-threaded programming.  It’s also been ported to FreeBSD (Hi Robert) so there’s really just no need to re-invent this wheel.   The source code is there, it works, just make it a subset of your library.  Its dependencies are deliberately very small because we wanted it to be low-level and cross-platform.

> 9. A lot of other things to be done, such as unified I/O streams which
> provide compression/transformation filters, protocols, math library, etc.

Look at GCD’s async I/O functions and, if you want to create transformation/compression filters, you really can’t do better than Security Transforms as a model.  Don’t let the name fool you - they were created for cryptographic tasks, but they’re really a generic, multithreaded, data pipelining model.  If I had those to do over again, I would not have had them called SecTransforms at all but merely added a Transform API to GCD and then built SecTransforms on top!

The rest of your proposal looks fine to me - date functions, unicode, etc - all pretty standard stuff.  Like I said, I have pretty recent experience with this with Core Foundation (see also OpenCFLite) and it’s all doable and very useful, you just need to remember all the caveats in my first paragraph! :-)

- Jordan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 236 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20140226/6f144f30/attachment.sig>