Gmirror rebuilding

[John Nielsen]

On Friday 31 July 2009 10:54:33 Willem Jan Withagen wrote:
> I lost one of my disk in a gmirror, so I inserted a fresh one.
> And thusfar things went rather smoothly,it started rebuilding
> automagically.
>
> That's good,but what isn't is:
>
> Jul 31 16:43:15 www kernel: ad2: FAILURE - READ_DMA
> status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=16344448
> Jul 31 16:43:15 www kernel: GEOM_MIRROR: Synchronization request failed
> (error=5). mirror/mirror[READ(offset=8368291840, length=131072)]
> Jul 31 16:43:40 www kernel: ad2: FAILURE - READ_DMA
> status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> LBA=16910976
> Jul 31 16:43:40 www kernel: GEOM_MIRROR: Synchronization request failed
> (error=5). mirror/mirror[READ(offset=8658354176, length=131072)]
>
> and ad2 is the original disk. So somewhere I'm left with corrupt files.
>
> And what's worse, once this happens, geom_mirror does not continue with the
> remainder of the disk...
> It claims it is, but there is no activity at all on the disks.
>
> So what to do????
>
> Hard way out would be to make a backup, reinstall the basic system with
> another/second fresh harddisk, and recover the backup.
> But that is a lot of work.

If you have a backup already then restoring from it would be the best option. 
If you don't then the "hard" way above is a good second (assuming you can 
read enough of your remaining disk to get your backup tool to cooperate).

You should make a backup in any case, but if you want to try to avoid 
reinstalling you could do some dd trickery. Remove the new disk from the 
mirror. Create a new mirror containing only the new disk. Run
	dd if=/dev/mirror/<original> of=/dev/mirror/<new> conv=noerror,sync

This will take a long time with the default block size (512 bytes, one 
sector), but the plus side is that you only lose the data from the sectors 
that cannot be read. Depending on the extent of the damage to the original 
disk and your level of desperation, you may want to make note of which 
sectors fail to copy on the first run and try to copy them again (dd if=... 
of=... skip=NN seek=NN count=1). See also sysutils/ddrescue, 
sysutils/recoverdm and similar (I haven't used any of them).

If you get a (mostly) viable clone, run fsck -f on it, assess the damage, 
update your fstab(s), reboot and add a second new disk to your new mirror.

> Why doesn't geom_mirror continue with the remainder of the disk?

I'll leave this question for someone else, but I suspect the behavior is 
intentional. As you say, you now have corruption on your volume so the best 
recourse is to restore from a known good backup.

JN