CFD: XMLification of NOTES

[Peter Wemm]

On Tuesday 30 March 2004 03:24 pm, Alexey Zelkin wrote:
> On Tue, Mar 30, 2004 at 05:05:24PM +0200, Dag-Erling Sm?rgrav wrote:
> > Alexey Zelkin <phantom at FreeBSD.org> writes:
> > > This weekend I have got some spare time (due to illness) and in
> > > order to get some rest from current tasks decided to pass thru
> > > old TODO file.  Most interesting task got my attention was old
> > > item related to XMLification NOTES.
> > >
> > > Important advantage of this step (IMO) is to provide possibility
> > > to check dependency/conflicts of kernel configuration file to end
> > > users. [...]
[...]
> > Dependencies between loadable drivers and subsystems are documented
> > in the source code (MODULE_DEPEND), though that information is not
> > used at compile time.  Dependencies that involve non-loadable
> > drivers and subsystems are not documented anywhere (except
> > sometimes in manual pages).  It would not be hard to modify
> > config(8) to obtain that information from e.g.
> > src/sys/conf/depend*. 

These are important points.  The runtime dependency information is 
encoded in the source in order to allow for free-form packaging of 
objects that self-describe their dependencies.  We've learned the hard 
way that having information duplicated in multiple places leads to 
things getting out of sync.

For what its worth, I personally find raw XML to be visually offensive.  
Machine readability ease doesn't make up for the developer pain in 
trying to edit/maintain it in this form.  I personally would avoid it 
like the plague.  (And thats an interesting question to ask..  Who are 
the people that you're expecting to maintain the XML format files?)

As a side note (but related), I've just finished the first part of a 
refurbished module loader.  I had to do this for amd64, but it has 
benefits elsewhere.  The main change is switching from 'cc -shared' 
format .ko files to 'ld -r' (relocatable) .ko files.  The big 
consequence of this is that it becomes trivial to build a kernel out 
of .ko files.  (Or at least embed .ko files into the kernel). 

The new module loader stuff allows this sort of thing:
hammer# cd /boot/kernel
hammer# ld -r -o usbstuff.ko u*.ko
hammer# ls -l usbstuff.ko
-rw-r--r--  1 root  wheel  662202 Mar 31 01:12 usbstuff.ko
You can then kldload the usbstuff.ko file and get all of the internal 
modules at once (I can't give an example of that part because I just 
paniced the machine :-] ).  Anyway, you can't do this with the 
present .ko format.

To take full advantage of this flexibility, I'd planned to do a 
completely new config(8) that used an extended files.*/options.* file 
set.  I wrote group of named.conf-style parsers for it a few years ago 
but never did anything with them.  config(8) was replaced by a couple 
of standalone parsers that converted the human-readable tables into 
something that was easy for perl (it was in the tree at the time) 
scripts to process and generate the kernel and Module makefiles on the 
fly.  What I was working on was building everything into .ka files and 
assembling the kernel and modules according to the packing list in your 
config file. [No Bruce, it isn't April 1st here yet]  GENERIC included 
a list of things to go in the static kernel and the generic modules 
list.  All the double compiling went away.  I had a huge amount of work 
to do though.

The second part of the work I have planned is to enable the kernel to 
have dependencies.  eg: link the kernel without either the traditional 
atpc nor the acpi code and do some sort of HAL layer and select the 
best one at loader(8) time.  (It doesn't have to do it this way, but 
its an example of what could be done.  It makes the acpi startup code a 
lot less painful.)  This requires a bit of care and thought though.

But this leads to interesting questions..  How do you specify run time 
dependencies vs static kernel link dependencies?  Suppose if_foo.ko has 
symbol reference dependencies on miibus.ko.  There is no reason why you 
should be prevented from embedding if_foo.ko into the static kernel and 
loading miibus.ko at boot time to satisfy the static kernel's symbol 
dependencies.  miibus is a bad example, because it is a poorly behaved 
newbus citizen and makes symbol references that should be done with 
method calls instead.  We have runtime dependencies that do not involve 
symbols.  newbus device stacks do not make promiscuous symbol 
references to each other, but can only function when all the components 
are present.  For example, you can have a device driver with pci, isa, 
eisa, cardbus and pccard attachments, all in the kernel.  It does *not* 
need eisa, cardbus, pccard and pci to be present in order to use it on 
an isa-only system.  You could concievably load eisa *after* the driver 
example above, and it all automagically works.  If anybody breaks this, 
I'll personally hunt them down and shoot them.

Anyway, perhaps the thing for me to do is find the parsers and scripts 
that I wrote a few years ago and freshen them up for some demos.
-- 
Peter Wemm - peter at wemm.org; peter at FreeBSD.org; peter at yahoo-inc.com
"All of this is for nothing if we don't go to the stars" - JMS/B5