[RFC] shipping kernels with default modules?

[mdf at FreeBSD.org]

On Sat, Jun 11, 2011 at 1:43 PM, David Schultz <das at freebsd.org> wrote:
> On Sat, Jun 11, 2011, mdf at freebsd.org wrote:
>> On Sat, Jun 11, 2011 at 10:18 AM, David Schultz <das at freebsd.org> wrote:
>> > On Sat, Jun 11, 2011, Adrian Chadd wrote:
>> >> Hi guys,
>> >>
>> >> Has there been any further thought as of late about shipping kernels
>> >> with modules only by default, rather than monolithic kernels?
>> >>
>> >> I tried this experiment a couple years ago and besides a little
>> >> trickery with ACPI module loading, it worked out fine.
>> >>
>> >> Is there any reason we aren't doing this at the moment? Eg by having a
>> >> default loader modules list populated from the kernel config file?
>> >
>> > I've been doing this for years, and it has come in quite handy.
>> > For instance, when my if_msk gets wedged, the only way to fix it
>> > short of rebooting seems to be reloading the driver.
>> >
>> > One issue, however, is that the boot loader is horrendously slow
>> > at loading modules.  (Either that or my BIOS has a braindead int 13h
>> > handler.)  Most of these modules aren't actually needed until much
>> > later in the boot process, so a mechanism to load non-essential
>> > modules after the file systems are mounted might provide a good
>> > solution.
>>
>> Indeed, at $WORK we're trying to get shutdown -> restart under 2
>> minutes.  Several seconds of this is moving things *into* the kernel
>> that need to be there (disk drivers), and everything else to a point
>> in init where modules can be loaded in parallel, using the faster disk
>> driver, rather than in serial with slow BIOS handlers.
>
> Have you found that drivers can be reliably loaded in parallel
> these days?  I'm always waiting for timeouts on four card readers
> and two optical drives, so that would be a big win for me.  IIRC,
> nothing can happen in parallel during boot because the scheduler
> is initialized very late in the process.  I'm not a device driver
> person, but I imagine there might be other assumptions that might
> get in the way as well.

With the specific set we have at $WORK it seems fine.  IIRC, Giant is
held when the module's init function is called, but it's dropped
during sleep so there's still some parallelism that can be achieved --
I would guess most driver init is essentially instant on fast modern
CPUs except when waiting for device probes, etc.

Cheers,
matthew


>> > OS X has an interesting solution, intended to preserve the
>> > flexibility of dynamic modules, while minimizing boot time.
>> > It provides a kextcache utility, which packages the kernel
>> > and all of the needed modules into a single binary for better
>> > locality on disk.  Unlike recompiling the kernel, running
>> > kextcache is fast, and the system runs it automatically when
>> > hardware or driver changes necessitate it.
>>
>> This would be interesting -- isn't it essentially a re-link of the
>> kernel + modules?
>
> They actually go further than that and "pre-bind" the modules,
> essentially doing the work of the dynamic linker ahead of time.
> I have doubts that this trick saves much time.  It also doesn't
> address Doug's concern about the overhead of the additional level
> of indirection needed for dynamic symbols; however, that overhead
> probably isn't significant either.
>