Devices disappeared after drive shuffle - or - how to recover and mount a slice with UFS partitions.

[Danny Carroll]

 I had a strange problem this week and although it's already solved, I
am still not sure what went wrong.
This is a long email so bear with me ;-)    There is a nice recovery
procedure at the end and a few questions along the way for someone who
knows more than I.

I have 15 drives attached to my home fileserver.  
    12 are connected via an areca controller (configured as
pass-through).   Data drives.
    3 are connected directly to the motherboard SATA controllers.    OS
Drives.

The motherboard is a Supermicro X7SBE with 8Gb RAM.

The 3 OS drives are 80Gb seagate sata drives and up till this week one
of them was running 7.3-Stable.
I wanted to reinstall so I decided to pull the OS drive, re-install 8.1
on one of the other 2 drives and use PJD's procedure for ZFS mirrored
root. 
(http://blogs.freebsdish.org/pjd/2010/08/06/from-sysinstall-to-zfs-only-configuration/)

I installed on ad6 and mirrored to ad4.  Ad8 was the old 7.3 drive which
was pulled from the server to make sure I did not accidentally overwrite it.

Install went fine and I was able to plug the old drive back in after the
upgrade and see the old 7.3 data (it was again ad8).  My problem arose
when I decided to test if the ARECA passthrough drives are really 100%
passed through.   The 12 drives are configured in a zfs raidz array.   I
use GPT partitions with labels that are used.
I powered down, pulled one of the drives (da0) and plugged it into one
of the SATA ports.

ZFS came up perfectly and a scrub revealed no errors.   This tells me
that a drive connected to an Areca controller and configured as a
pass-through is no different to a drive connected to a standard SATA
port.   This is good to know if my areca card ever fails.  (Areca
Pass-through differs from JBOD in that JBOD ignores the Areca's cache
(and battery backup) and passthrough does indeed use the cache.)

The da0 drive is now ad4, the 2 OS drives are now ad6 and ad8 and the
old OS drive (which I plugged back in after the upgrade) is now ad10.

All looked good but the 2 slices on the ad10 drive did not show up in
/dev/.   My data was on the second slice with 4 UFS partitions on that
slice.
Running fdisk on ad10 showed the slices were there, but just not in /dev/.

I had backups but I wanted the data anyway so I used testdisk to create
an image of the second slice.   (dd would have worked just as well I guess).
I then mounted the raw slice via a md0 device.   As expected md0a, md0d,
md0e and md0f showed up in /dev/
Great...  I had my data back!

My only real question is.   Why did the devices fail to be created in
/dev from the original disk?  And is there a way to force devfs to
rescan a device and create those sub-devices?

And now, a procedure for anyone who finds themselves having to recover a
slice containing UFS partitions.  I kinda guessed my way though this and
was surprised that it was so easy.
    Use sysutils/testdisk to dump the partition to a flat file.
    mdconfig -a -t vnode -f /data/image.dd -u 0
    mount /dev/md0a /oldserv
    mount /dev/md0f /oldserv/usr
    mount /dev/md0d /oldserv/var
    mount /dev/md0e /oldserv/tmp
    done!


Interestingly enough, I tried this with a zvol as well.
    Use sysutils/testdisk to dump the partition to a flat file.   Called
image.dd  (it was about 70G)
    Create the zvol:  zfs create -V 70G data/oldserv  (make sure size is
the same)
    dd if=/data/image.dd of=/dev/zvol/data/oldserv bs=1M


Unfortunately it did not work out.   The slices were created but I had
trouble mounting them.  I am not sure why.

I learned a little about UFS data recovery, a little about md0 and a
little about zvols today.
I assume if I had have done the same with the whole disk device instead
of the slice, I would have seen md0p2a md0p2d etc in /dev/...

-D