Booting gvinum fails: missing /dev/gvinum/* device entries

[Aaron Gifford]

I'm running 6.1R (amd64) on a system, and recently added more drives.  I
decided I would eventually migrate my root system off the existing drive
(/dev/ad0a) to use a gvinum root (/dev/gvinum/root) filesystem and
eventually remove the old hard drive altogether.

The two added drives (ad1 and ad2) are labeled (with bsdlabel) identically
like this:

#       size   offset    fstype
a:   4194039      281    4.2BSD
b:   4194394  4194320     vinum
c: 625142448        0    unused
d:   4194304       16     vinum
e: 616753824  8388624     vinum

The gvinum configuration looks like:

# Drives for the root gvinum RAID-1 volume:
drive a1 device /dev/ad1d
drive a2 device /dev/ad2d
#
# Drives for the swap gvinum RAID-1 volume:
drive b1 device /dev/ad1b
drive b2 device /dev/ad2b
#
# Drives for the usr gvinum RAID-1 volume:
drive c1 device /dev/ad1e
drive c2 device /dev/ad2e
#
# RAID-1 volume for root:
volume root
  plex org concat
    sd length 0 drive a1
  plex org concat
    sd length 0 drive a2
#
# RAID-1 volume for swap:
volume swap
  plex org concat
    sd length 0 drive b1
  plex org concat
    sd length 0 drive b2
#
# RAID-1 volume for usr:
volume usr
  plex org concat
    sd length 0 drive c1
    sd length 0 drive c2

Once the gvinum volumes were created, new filesystems created, and mounted
temporarily on /mnt and /mnt/usr, I added a line to loader.conf in the new
drives' future /boot directory containing 'geom_vinum_load="YES"'.  Oh, and
the new drives' future /etc/fstab contains:

# Device         Mountpoint FStype  Options Dump Pass#
/dev/gvinum/swap none       swap    sw      0    0
/dev/gvinum/root /          ufs     rw      1    1
/dev/gvinum/usr  /usr       ufs     rw      2    2

Oh, boot blocks were also written to both drives before the gvinum volumes
were set up using bsdlabel -b.

Now for the test.  I rebooted, changing the boot order in BIOS so the first
drive, ad1 (instead of ad0) was the boot source.

THE PROBLEM:

No matter what I tried, booting always failed at the point of mounting the
root filesystem.  I would be kicked to a "mountroot>" prompt.  Any attempts
at mounting root from ufs:/dev/gvinum/root failed.  Even if I loaded
geom_vinum.ko by hand during boot to be absolutely sure gvinum was loaded,
this failed.  At "mountroot>", the gvinum devices did not show up at all.

So I fell back and mounted /dev/ad0a (my old non-gvinum drive) as root.
That worked.  kldstat showed that indeed geom_vinum.ko WAS successfully
loaded.  But when I looked in /dev nothing for gvinum showed up there.  It
was empty.

The "gvinum list" command showed all my gvinum volumes working, but there
were NO entries in /dev.

No wonder I was unable to mount /dev/gvinum/root -- There was no device
entry.

So what's the deal?   Any suggestions?   Is there something I need to
compile in my kernel?

If I boot off my old drive and only load the geom_vinum.ko module by hand at
the command-line by using the gvinum command, the entries in /dev/ DO appear
normally.

Puzzled, frustrated,
Aaron out.