Should Xen be a sub-arch or a build option?

[Warner Losh]

> > Let me say in advance that this is not an invitation to discuss the
> > technical merits of xen. This is purely a request to discuss how one
> > would structure the tree were one to import it into CVS.
> > 
> > Hypothetically speaking, if one were to import Xen support into CVS
> > what would be the best way to go about it?
> > 
> > There are a number of choices when doing it as a sub-arch:
> >   - A separate directory for i386 and amd64
> >       - sys/xen-i386
> >       - sys/xen-amd64
> >  - A shared directory as most of the bits will be shared:
> >       - sys/xen          - common bits
> >       - sys/xen/i386   - i386 specific bits
> >       - sys/xen/amd64 - amd64 specific bits
> > 
> 
> If most of the bits will be shared, then lets share them, that is, use
> the second proposal.  I couldn't find a good example in the tree for a
> precedent, though the powerpc might come close, with its powermac and
> powerpc sub-directories.

The embedded stuff has a lot of different 'ports' living in one
directory currently.  This is both good and bad.  For arm, we can run
on a number of different SoCs/boards and we use the config system to
pull in the right bits.  If you look in detail, the way this is done
is to effectively have an arm base that provides most of the standard
stuff, and then the kernel config for each board/class of processors
pulling in standard stuff and optional stuff.  It works OK.

The bad part is that it totally glosses over softfloat vs FPU, endian,
processor level, ABI, etc.  Since that's not relevant for xen, I'll
just mention it in passing.

The basic thing that should determine if there's a separate
architecture or just a build option is how much of the *machdep files
change.  If vm_machdep is totally different, then it needs to be a
separate architecture (or subport ala arm/powerpc).  If it is just a
few ifdefs in a couple of places to do extra thing, assume certain
things, etc, then it could be as little as a build option.

pc98 goes the full architecture route.  However, it does a reach-over
to grab shared code.  This works out well in practice, since there are
kernel <-> userland interface differences ebtween pc98 and i386.  The
number of these is small (mostly relating to video modes and disk
layout issues, iirc), but enough that this arrangement works well.  If
XEN has a lot of userland/kernel differences from i386/amd64, then it
would make more sense to have as its own architecture.  However, the
config tools depend on the architecture we return for what make calls
MACHINE and MACHINE_ARCH, mostly the latter.  a xen/i386 or a
xen/amd64 would just be changing MACHINE not MACHINE_ARCH, so likely
wouldn't break too many things.  Just like pc98/i386 doesn't break
things.

One wrinkle, however, is that presently MACHINE uniquely defines
MACHINE_ARCH.  If one wanted xen for both i386 and amd64 variants,
there would need to be some infrastructure work.  So that argues
against it.  Also, if one were doing a default build on a xen/i386
machine, our current build tools would build a xen/i386 system, not a
i386/i386 system.  Maybe that's desirable, maybe not.  If not, then
that argues against going the pc98 route.

> > It could, in principle, also be done as a build option. I'm not sure
> > how well it would mesh with the existing build tools as there are a
> > number of files that I would not want to compile in (e.g. code that
> > talked directly to the BIOS) that is normally built by default. In
> > that case I would structure it:
> > 
> >       - sys/i386/xen     - xen specific bits for i386
> >       - sys/amd64/xen - xen specific bits for amd64
> > 
> > There is also a question of where the drivers should be put. I propose
> > that they would be put under sys/dev/xen, so you would have e.g.
> > sys/dev/xen/xennet, sys/dev/xen/xenblk etc.
> 
> Yes, this makes sense too.

Having it be a build option wouldn't preclude having extra bits being
pulled in for that build option.  If there were extra drivers that
were needed, then it is cool.

>From what I've seen of xen, one could make an argument for any of the
three methods that we presently use in the tree.  All of them would
have their pros and cons.  The stongest being for build option,
followed closely by subarch.

If it were me, I'd look at having it be a build option, much like PAE
is a build option.  PAE has a bigger impact on the i386 world than
xen, and it is only an option.  PAE breaks kernel ABIs, while xen
doesn't (as far as I know).  PAE changes the size of things like
vm_addr_t and bus_addr_t.  I think this would fit best with our
current world view and sensabilities.

Just my take.  Maybe there are arguments for another option that would
push it in another direction, but from what I've seen so far I'm
unaware of them.

Warner