Booting gvinum fails: missing /dev/gvinum/* device entries

[Aaron Gifford]

On 8/26/06, Aaron Gifford <astounding at gmail.com> wrote (or in other words, I
wrote):

> No matter what I tried, booting always failed at the point of mounting the
> root filesystem.  I would be kicked to a "mountroot>" prompt.  Any attempts
> at mounting root from ufs:/dev/gvinum/root failed.  Even if I loaded
> geom_vinum.ko by hand during boot to be absolutely sure gvinum was loaded,
> this failed.  At "mountroot>", the gvinum devices did not show up at all.
>
> So I fell back and mounted /dev/ad0a (my old non-gvinum drive) as root.
> That worked.  kldstat showed that indeed geom_vinum.ko WAS successfully
> loaded.  But when I looked in /dev nothing for gvinum showed up there.  It
> was empty.
>
> The "gvinum list" command showed all my gvinum volumes working, but there
> were NO entries in /dev.
>
> No wonder I was unable to mount /dev/gvinum/root -- There was no device
> entry.
>
> So what's the deal?   Any suggestions?   Is there something I need to
> compile in my kernel?
>

I managed to fix my own problem.

Before I posted, I did some searching and found previous posts of people who
were missing /dev/gvinum device entries.  They mentioned that it was corrupt
GEOM vinum metadata.

I did not believe that this applied to my situation, as while booted of my
older working i386 kernel running on a non-GEOM-vinum drive, the GEOM vinum
volumes appeared and worked normally.  Therefore, I concluded, the metadata
must be in working order.

However, no matter how I tried to boot off the amd64 kernel on the gvinum
root volume, it failed.  Closer examination made it look like perhaps under
the amd64 kernel and geom_vinum.ko module, perhaps the metadata stored in
the first 265 sectors (512-byte sectors) of the vinum partitions was being
interpreted differently than it was by the i386 kernel and geom_vinum.ko
module.

This leads me to believe that the problem was one of the following:

1) Due to my changing the BIOS boot order of my drives (so as to boot from
the gvinum drive running amd64 instead of the non-vinum i386 kernel drive),
perhaps GEOM vinum got confused about the configuration since the BIOS
device boot order was different; or

2) because of differences between the i386 and amd64 architectures, GEOM
vinum volumes created under one architecture do not correctly appear under
another architecture.

If #1, then I need to remember this and be careful about changing BIOS boot
order while running GEOM vinu.

If #2, then I believe this is a bug in GEOM vinum.  Volume metadata should
be cross-architecture independent.

So, FreeBSD gurus wiser in the innards of GEOM vinum than this user (pretty
much everyone who knows how GEOM vinum actually works), which is the cause
of my woes?  Is it #1?  Or is it #2?  Or am I missing a third possibility
entirely?

Thanks for any future enlightenment!

Aaron out.