Problems with dump and restore

Tue Aug 12 19:30:16 UTC 2014

Hi Karl -- thanks for your thoughts and response.

On Tue, 12 Aug 2014, Karl Vogel wrote:

>> I have used dump/restore for several years, very happily, to maintain
>> backups on my machine.
>
>  Just to be clear: you've *restored* from dumps before and had no problems?

Yes, on several occasions, usually when changing disk layout or structure.
The most recent occasion was at least six months ago (possibly a bit more),
likely using FreeBSD 9.2.

Being used for structural changes, I therefore normally do a zero level
dump, and then need only a zero level restore.  I have had a couple of
disks go south on me in the last years, however, and on those occasions
did several restore levels to recover the data on multiple partitions
that were on the dead disk -- with absolute success.

>> I wanted to restore the /usr partition to the state it was in at the last
>> (level 5) backup.  My expected steps to achieve this are:
>> 	o create a replacement filesystem on the drive:
>
>  Did you use the same newfs options to create the original /usr partition?
>  I don't have a FreeBSD box in front of me right now, so I'm grasping at
>  any available straw here.

Yes; even verified the options using dumpfs -m to ensure that the setup
was exactly the same as what I had in my logbook.

>> 	o restore the level 0 dump
>> 		restore ruf /backup/dumps/current/usr.dump
>> 	* this is the first sign of trouble, as restore output the warning
>> 		expected next file 19266003, got 19100935
>
>  Can restore show a table of contents for that dump, so you know what files
>  correspond to those inodes and where it might be crapping out?

Sure -- from the table of contents, inodes 19266003 refers to
 	./local/share/licenses/patch-2.7.1/catalog.mk
while inode 19100935 is not listed in the dump at all.

>  Can you restore any of that level-0 dump at all?  If so, can you restore
>  single files or some parts of the tree, like /usr/local?  If it's expecting
>  a file and finds a directory in the dump instead, maybe that's something
>  you could fix by hand before restoring in a piecemeal fashion.

The "expected next file" error is the only error from the level 0 restore.
As far as I can tell, everything except for that single file has come back
fine, from the level 0 dump.

In the time since I sent the first message, I have booted the machine
back to multiuser (the zero level is from July 11, so close enough that
I can stagger along for a little bit), and re-run the restore on a scratch
volume that I have available.  Progressing to the level 2 restore (there
having been no level 1 dump performed), the complete set of error messages
from both restores are:
 	+ restore ruf /backup/dumps/current/usr.dump
 	expected next file 19266003, got 19100935
 	+ restore ruf /backup/dumps/current/l1d0/l2d0/usr.dump
 	./src: (inode 10032000) not found on tape
 	./ports: (inode 14205312) not found on tape
 	bad entry: removeleaf: not a leaf
 	name: ./local/share/licenses/openldap-client-2.4.39
 	parent name ./local/share/licenses
 	sibling name: ./local/share/licenses/RSTTMP03130448
 	next entry name: ./local/share/licenses/openldap-client-2.4.39/OPENLDAP
 	entry type: NODE
 	inode number: 3613621
 	flags: NIL
 	abort? [yn] y
 	dump core? [yn] n

None of the inodes listed in the error messages are in the output from
the table of contents.

After attempting the level 2 restore, the top-level directory of the
recovered filesystem looks like this:
 	.snap           games           lib32           obj             swap
 	RSTTMP010032000 home            libdata         restoresymtable tests
 	RSTTMP014205312 include         libexec         sbin
 	bin             lib             local           share

The RSTTMP* directories contain additional RSTTMP* entries for unnamed
found cruft, but some recognizable files:
 	# ls RSTTMP01*
 	RSTTMP010032000:
 	COPYRIGHT               RSTTMP010192614         RSTTMP010834806
 	LOCKS                   RSTTMP010272816         RSTTMP010915023
 	MAINTAINERS             RSTTMP010272822         RSTTMP011075378
 	Makefile                RSTTMP010272857         RSTTMP011075564
 	Makefile.inc1           RSTTMP010272860         RSTTMP011235981
 	ObsoleteFiles.inc       RSTTMP010353048         RSTTMP011236023
 	README                  RSTTMP010353178         RSTTMP011236050
 	RSTTMP010032001         RSTTMP010353195         RSTTMP011236093
 	RSTTMP010112261         RSTTMP010513687         UPDATING
 	RSTTMP010192569         RSTTMP010834762

 	RSTTMP014205312:
 	.arcconfig      RSTTMP014205917 RSTTMP014207366 RSTTMP014447559 RSTTMP015170651
 	.gitignore      RSTTMP014206111 RSTTMP014207415 RSTTMP014606720 RSTTMP015248825
 	CHANGES         RSTTMP014206140 RSTTMP014207416 RSTTMP014687196 RSTTMP015249051
 	CONTRIBUTING.md RSTTMP014206371 RSTTMP014207431 RSTTMP014687230 RSTTMP015250173
 	COPYRIGHT       RSTTMP014206656 RSTTMP014207509 RSTTMP014688864 RSTTMP019341927
 	GIDs            RSTTMP014207111 RSTTMP014207550 RSTTMP014689162 RSTTMP03290912
 	INDEX-10.db     RSTTMP014207165 RSTTMP014207557 RSTTMP014768653 RSTTMP03531831
 	INDEX-7         RSTTMP014207172 RSTTMP014207580 RSTTMP014768663 RSTTMP03531837
 	INDEX-8         RSTTMP014207196 RSTTMP014207597 RSTTMP014848208 RSTTMP03532331
 	INDEX-9         RSTTMP014207212 RSTTMP014207598 RSTTMP014848898 RSTTMP03532533
 	INDEX-9.db      RSTTMP014207266 RSTTMP014207602 RSTTMP014849014 RSTTMP03534719
 	MOVED           RSTTMP014207267 RSTTMP014366176 RSTTMP014928591 RSTTMP03692298
 	Makefile        RSTTMP014207277 RSTTMP014367788 RSTTMP015088613 RSTTMP03932616
 	README          RSTTMP014207318 RSTTMP014368109 RSTTMP015088615 RSTTMP05538024
 	RSTTMP014205313 RSTTMP014207329 RSTTMP014446601 RSTTMP015088621 RSTTMP07865164
 	RSTTMP014205612 RSTTMP014207338 RSTTMP014446815 RSTTMP015089051 UIDs
 	RSTTMP014205756 RSTTMP014207341 RSTTMP014447545 RSTTMP015169454 UPDATING

The first entry appears to be data for /usr/src, while the second is
/usr/ports, which makes sense given the error messages and the recent
security updates that have touched both of these directories (so the
fact that they are changed since July 11 makes sense).

>> Is there any means of validating the dump file, other than the -N
>> option (which returns no warnings on any of these files)?
>
>  That's one reason I got away from the Solaris version, which had no
>  problem lying to me about success or failure.  All I can think of is to
>  try restoring the entire dump under /tmp (or someplace trashable that won't
>  mess up your system) and look for errors before you trust the dumpfile.

I suppose I am in agreement -- but also in shocked amazement that dump
and restore can be broken without the sky falling.

>  I remember using dump to quickly get me a list of files that have changed
>  since the corresponding /etc/dumpdates entry, and then making a backup
>  using pax or cpio.  Maybe that would work for you.

I'm not sure that I understand your suggestion here -- are you suggesting
this as a future backup strategy as a replacement for dump levels higher
than zero?  Perhaps I am missing something here.

>  Feel free to repost if you think it's worth the bandwidth.  Good luck.

Doing so, in case anyone else can weigh in with any further ideas.  Thanks
for the suggestions and ideas.

I suppose I will now open a bugreport, too, as it does seem that there
is something not right.

Andrew.