File system corruption upon reboot with gmirror

[Gunther Mayer]

Hi guys,

I recently updated my FreeBSD 6.3 on our server to the latest patch with 
freebsd-update and seeing that it involved some kernel patches on 64bit 
I had to reboot. So I carried out an automated reboot during low-load 
times but alas, the box never came back up again.

After gaining physical access to the console I realised that it choked 
on the unclean /usr file system and was unable to proceed as the 
automatic fsck failed, prompting for an emergency shell. An fsck -y 
followed by a reboot sorted out the issue but it caused a good 1.5h of 
total downtime which should have been only 4min.

So, why was the file system unclean even though I rebooted properly?

Afaic this only happens on a power loss or otherwise unclean shutdown 
but I used the "reboot" command from the shell (in a background (sleep 
21600; reboot) & but that shouldn't matter). So surely it would have 
flushed all the buffers in time? Or is the standard 60 seconds it waits 
maximum for kernel tasks to finish upon reboot too low and it couldn't 
finish in time (in which case, how do I change that?)?

To give you a bit more background, I run a gmirror(8) RAID 1 over two 
disks whose health seems intact (zero bad gmirror log entries):

$ mount
/dev/mirror/gm0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/mirror/gm0s1e on /tmp (ufs, local, soft-updates)
/dev/mirror/gm0s1f on /usr (ufs, local, soft-updates)
/dev/mirror/gm0s1d on /var (ufs, local, soft-updates)

Gunther