ataraid's revenge! (Was: Re: A nasty ataraid experience.)

[Rick C. Petty]

This question should have been directed to freebsd-geom.

On Thu, Jul 23, 2009 at 03:42:47PM -0500, Sean C. Farley wrote:
> 
> Anyone know if I can boot off of a gvinum partition and/or how it works 
> (or does not) with various label schemes?

http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/vinum-root.html

There are some downsides.  First of all, when you mount your root partition
under vinum, you can't unload the geom_vinum module (for obvious reasons).
If you decide to relocate volumes in the future, you won't be able to
remove the disks from the vinum.  The reason is because you're not allowed
to change the bsdlabel if there's a vinum slice while vinum is loaded, and
you cannot remove a "device" from vinum even if it has no subdisks.  I know
lulf@ was looking into the second part.

I used to do this on all my machines.  Sometime soon after 7.1-release, I
noticed a significant slowdown in disk performance, perhaps due to disk
scheduling.  Basically while any moderate I/O hit one gvinum volume, all
other I/O on that disk was essentially suspended until the the heavier I/O
completed.  What this meant was that if I did an rsync to or from a volume
whose subdisk resided on the same physical disk as my root and /usr
partitions, I could not run any commands until the rsync finished or was
killed (with control-C, since I couldn't even get a "killall" to execute).
After that I switched all my machines to use gmirror which helped slightly.
During that switch is when I discovered you need to be pretty clever to go
away from "vinum root".

I don't mean to say not to use gvinum.  I put a lot of time into it,
submitted some patches, and did a lot of testing with Ulf's deltas.  There
were still a couple of bugs when I stopped using it, but that doesn't mean
they won't be fixed by 8.0.  I just couldn't deal with the I/O performance
issues which I was never able to track down.  I need my workstations to be
responsive while I'm rsyncing stuff around, and with 4 cores and 8 GB of
RAM, I expected it to behave better.  YMMV.

FYI, the trick I used when migrating mirrors in vinum to gmirror:
1). Pick one disk to convert to gmirror first.  I'll call this disk0.
2). Remove all subdisks which are attached to the device using disk0.
3). Reboot into single-user with vinum unloaded, specifying the raw device
root (from disk1) instead of /dev/gvinum/root-volume.
4). Wipe sector 8 of the vinum slice on disk0.
5). Create a gmirror with disk0, re-bsdlabel, create new filesystems, etc.
6). Load geom_vinum kernel module.  This may panic the kernel.  If so,
reboot normally using the vinum root volume.
7). Mount the new filesystems.
8). Mount the volumes you want moved to the mirror and rsync them over to
the filesystems mounted on disk0.
9). Unmount all vinum volumes and try to unload the geom_vinum module.  If
this panics the kernel, or if there was a panic in step 6, you'll need to
reboot into single-user again with geom_vinum unloaded.
10). Wipe sector 8 of the vinum slick on disk1.
11). Add disk1 to the gmirror, which should start a sync.
12). Load the geom_vinum module again, if you have other volumes not part
of this mirror.  If this panics the kernel, or if the kernel panicked after
step 11, a normal restart might be successful.

-- Rick C. Petty