Problems with dump and restore

[Andrew Hamilton-Wright]

I was attempting to restore my /usr partition today, and have encountered
some rather terrifying issues using restore.


Some background ...

I have used dump/restore for several years, very happily, to maintain
backups on my machine.

I have a level 0 dump of each file system, and then a cron-based script
that does higher level dumps on a regular basis.  I therefore have dumps
at the following levels for this filesystem at the moment:  0, 2, 3, 5

These were created using snapshots, so the level 0 was created via
 	dump 0uLCf 32 - /usr
and higher level dumps were created similarly.

My uname info is:
 	FreeBSD qemg.org 10.0-RELEASE-p7 FreeBSD 10.0-RELEASE-p7 #0: Tue Jul  8 06:37:44 UTC 2014     root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64
I wanted to restore the /usr partition to the state it was in at the last
(level 5) backup.  My expected steps to achieve this are:
     o go to single user (I did this through a full reboot)
 	o create a replacement filesystem on the drive:
 		newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 \
 				-g 16384 -h 64 -i 8192 -k 0 -m 8 -o time \
 				-s 415236096 /dev/ada0e
 	o mount the drive as /usr, and change directory to the mount point
 	o restore the level 0 dump
 		restore ruf /backup/dumps/current/usr.dump
 	* this is the first sign of trouble, as restore output the warning
 		expected next file 19266003, got 19100935

 	o restore the level 2 dump
 		restore ruf /backup/dumps/current/l1d0/l2d0/usr.dump
 	* this failed, indicating that the restore was corrupt (unfortunately
 	  I do not have the full text of the errors received, but a complaint
 	  that an entry was "not a leaf" was in the first message)

Frankly, this terrifies me.  If dump and restore cannot be trusted
as a robust backup solution, I don't know where to turn to.

Some questions then:
- is anyone else using dump/restore as their main backup method?
   Are you using snapshots?  If so, have you seen anything like this
   when running restore?

- is there any means of validating the dump file, other than the -N
   option (which returns no warnings on any of these files)?

- does anyone have any advice that may help determine what may have
   gone wrong?


Thanks,
Andrew.