vinum + powerfailure -> one volume corrupt, others unaffected

[Peter Schuller]

Hello,

I just had a "power outtage" (= i accidentally hit the killswitch on the 
master power source...). One of the machines affected was one running FreeBSD 
with several vinum volumes[1].

When booting up afterwards, all but one of the vinum volumes recovered 
completely. By "completely" i mean vinum shows the volumes as 'up' and fsck 
reported no problems.

One of the volumes however, did not quite survive (this is the 'backup_a' 
volume if you looked at the configuration). Vinum does report the volume as 
'up', however fsck reports a huge number of partially allocated inodes. I 
stupidly ran fsck -y on the volume before investigating further, and it 
wanted to remove/removed[2] basically all files on the partition[3].

Two things are however interesting. I performed a post-crash "vinum 
printconfig"[4]. Here's the diff relative to the pre-crash configuration[5] 
(minus the timestamp):

-drive backup_a_1 device /dev/ad1s1b
-drive backup_a_2 device /dev/ad0s1a
-drive diska device /dev/ad6s1a
 drive diskb device /dev/ad7s1a
+drive diska device /dev/ad6s1a
 drive diskc device /dev/ad5s1a
 drive diskd device /dev/ad1s1a
+drive backup_a_1 device /dev/ad1s1b
+drive backup_a_2 device /dev/ad0s1a

The two settings appear logically identical, but I find it very interesting 
that the one volume that happens to have become corrupted somehow has been 
sorted differently in the output; perhaps signfiying some underlying change.

Unfortunately I do not have a 'vinum dumpconfig' of the configuration as it 
appeared before the crash, but I put up a copy of the config as it appears 
now[6].

Interestingly:

scode-whitestar# fsck -n /dev/vinum/backup_a
fsck: Could not determine filesystem type

And sure enough, looking at the contents of the first part 
of /dev/vinum/backup_a, it's *ALL ZEROES*. The first non-zero information is 
somewhere beyond the first 200 (512 byte) blocks.

Does anyone know what might have happened? Did the volume offsets within the 
drive change perhaps? Anything else that sounds likely to explain the 
behavior? I get the feeling this wasn't just a random occurance.

This is on FreeBSD 5.2.1p9 (yeah I know 5.3 is out; I just haven't gotten 
around to upgrading yet, in part for fear of difficulties with the 
vinum->gvinum transition).

[1] Volume config: http://www.scode.org/vinum-list.txt
[2] Not sure which; i ran with -y but upon interrupting fsck and re-running it
*seemed* some things that were supposedly already fixed were still being 
reported.
[3] I don't know for a fact that it was *all* files, but based on everything 
that scrolled past it was at the least a pretty random set of files from the 
filesystem.
[4] http://www.scode.org/vinum-printconfig-postcrash.txt
[5] http://www.scode.org/vinum-printconfig-precrash.txt
[6] http://www.scode.org/vinum-dumpconfig-postcrash.txt

-- 
/ Peter Schuller, InfiDyne Technologies HB

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller at infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey at scode.org
E-Mail: peter.schuller at infidyne.com Web: http://www.scode.org