Compiler toolchain roadmap

Tue Apr 8 07:03:37 UTC 2014

On Apr 7, 2014, at 8:03 PM, Warner Losh <imp at bsdimp.com> wrote:

> This is a desire, not a hard requirement. As such, there may be missing bits if you chose to go this route. At least that’s been the notion in the past when using llvm features has come up. The notion for this path has always been ‘It is possible, but only a subset of the functionality may be available.”
> 
> So if one were to add blocks, it would need a knob WITHOUT_BLOCKS that would disable all functionality tied to them. For many applications, this is a reasonable subset (based on my guesses at how intrusive this would be to the system). And it might even be automatically selected based on compiler support, but that’s another can of worms.

I’m glad to hear that building with foreign compilers is more of a desire than a hard constraint.  I’m still not sure what the scenario would look like where libdispatch has been built WITHOUT_BLOCKS and is now essentially API incompatible with the libdispatch compiled WITH_BLOCKS (since you wouldn’t get functions like dispatch_async() but would get the dispatch_async_f() function which takes a function pointer/context variable instead), but I guess that’s a problem for the hypothetical wacky-compiler user to figure out?

You seemed to imply that when you said FreeBSD was already using a number of custom compiler extensions, I just wasn’t sure that this extended as far as “you can build FreeBSD, for the most part, with your weird-ass compiler but you can also expect it to be a different FreeBSD than what’s in base for x86_64 (for example) because various things will be switched off, API incompatible, or simply not compiled at all."

If that’s the case, then woohoo, when can I import libdispatch into base? :)  [I’m being semi-facetious here, I realize there are a few more steps to go].

> So to take this a step further… There’s many levels of integration here…  First, there’s the kernel, which is most often the bit of code people want/need the special compiler for. Next, there’s having the feature available in user land. Finally, there’d be a wide-scale integration of this feature. I see very few programs in base benefiting from libdispatch, honestly, but that doesn’t mean the set is empty. Do you have a longer write up on what you’d like to do here?

Sure, I can write something up as part of my “any objections to doing this?” step, but just to summarize briefly for this conversation:

1. Libdispatch does not need to touch the kernel at all, nor does the kernel require blocks support.  The pthread_workqueue() stuff that Stacey Son did all those years ago would be a nice optimization (this lets libdispatch balance its thread pool loads across multiple applications, not just within a single application) but it’s not strictly necessary, and it can also be added later without any consumer of libdispatch (henceforth referred to by its other name of GCD, because that’s easier to type!) needing to know or care.

2. Once we have GCD in base, I can then start looking at adding libnotify and its associated notifyd daemon, which is a generic system notification mechanism.

3. Once libnotify is in base, I’ll then add some basic notifications to Libc, like timezone changes, hostname changes, and so on, invalidating some of its internal caches in the face of said changes (there are some fairly expensive stat() calls you can eliminate once you have the far lighter-weight shared memory flag checking that notifyd provides, which also helps save power by not walking around the filesystem so much).  The notifyd service can also translate filesystem notifications into system notifications, which means services like samba and netatalk can detect (cheaply) when something has changed out from under them.

Beyond that, we probably need to talk about another daemon, which on OS X is called configd, which deals with larger scope configuration changes like network devices coming and going, default routes changing, and so on.  This does require some kernel up-call mechanisms, but it will “publish” its results in terms of notify(3) notifications so applications can subscribe to those events the same way.  If we can hack usbd into doing the same for USB events, that’s great, otherwise I would imagine configd dealing with that sort of hardware notification service as well.

> I see a continuum of answer here: If you want to modify the kernel extensive to use blocks, then that’s going to be a much bigger problem than having a few daemons and a library in the tree that require them which is a bigger problem than having a few daemons using it, but optionally, and a library which is a bigger problem than a library in the tree which is a bigger problem than the status quo.

I.. think… I follow that sentence enough to hope I’ve answered it.  No blocks in the kernel.  Nothing more than a few daemons and a library and, at some point, a way of figuring out when the kernel has made certain changes to the system configuration, if and where said changes can’t be intercepted and dealt with at the libc layer or otherwise noticed by a user-land daemon which can post the notifications for any subscribers of them.

As to the subscribers, they won’t have to use blocks unless they choose to consume one or more of notify(3)’s block APIs.  The notify mechanism was written with both old-style and new-style clients in mind.  If your daemon wants a signal when the notification it’s subscribing to changes, fine.  If it wants some data on a file descriptor which it’s poll(3)/kqueue(3)ing, fine.  See https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man3/notify.3.html and, of course, ignore the mach-specific delivery methods as N/A.

> And 8.1 for libdispatch? Is it really so portable it would work with our old, crappy gcc pre blocks update?

Sorry, misunderstanding.  I was referring to the original slashdot / phoronix article which stated that GCD would be coming to FreeBSD in 8.1.  Apparently, according to FreeBSD’s own wiki (https://wiki.freebsd.org/GCD) it works, too.

- Jordan