gvinum raid5 cannot find file system superblock after removing
a disk
Edward Sutton
mirror176 at hotmail.com
Fri Nov 16 01:04:05 PST 2007
Ulf Lilleengen wrote:
>Your description was a bit hard to understand, so please post a more detailed
>description of what you did, because all I got out from this was:
>1) You have a RAID-5 volume, and ad8 fails.
>2) You dd'ed the first 64 mb of ad8...
>3) You tried to repartition ad8... and then what? did you try rebuildparity?
>Could you also please post your gvinum configuration (gvinum printconfig) and
>listing (gvinum list)?
Hopefully word wrap does not kill the output of the configuration (at the end of this email). On a Promise TX4 SATA150 , I have 3 400GB drives which i partitioned (sliced actually if I recall my BSD titles correctly) in half. Seemed appropriate because I do not know of any utilities that let me manipulate the layout like I can with windows and linux pieces. If I do not like my layout, I can always try again on the other half; space is available to migrate back and forth.
On the first half, I setup a bootable system; Swap striped across the 3 disks, root mirrored across the 3 disks (yes, 2 copies. I could not think of a better idea for the leftover space on the 3rd disk than make another copy), usr var and tmp are striped with parity across the 3 drives.
I have had trouble with system crashes. One appeared to be drive related and the best I tracked from more recent crashes appeared to be with swap space; after I use so much, I'd get a crash. I have been using swap space from an unrelated hard drive and have achieved uptimes of>1 month again. Only 2 of the 3 entries for the root mirroring ever appear to work once something goes down and I cannot start it because its in use after booting (which it sounds will go away with future patches)
I was trying to move away from my vinum play to work a bit more with gmirror (which I setup a server with for my uncle). Instead of moving over to the ad*s2 parts of the disk, I figured I would just use all of one of the three disks. I had a disk that was currently out of sync (ad8), so I figured I would pull that disk out. I made a backup copy of the first 64MB with dd because I figured using fdisk only touches the start so I could put back any of those changes. I proceeded by rebooting from a FreeBSD copy on another drive (which does not load geom_vinum by default but does have it available) and repartitioned the drive to use the entire 400GB together. I did not format that space yet and after finidng I could not reboot my gvinum configured system, I restored the 64MB and found that I still could not boot the gvinum configured system. The root partition was okay (on 2 of the 3 disks as expected), but all raid5 disks errored out with the superblock message for both mounts and fsck.
I played with editing the on disk configuration which can be gathered by `dd if=/dev/ad4s1h skip=9 count=5` in the case of my configuration. I'd use my favorite text editor to change 'sd' lines to contain the plex and plexoffset and change it from 'down' to 'up' (and reversing that on other lines when disabling disks) and found that I was able to get other responses on partitions and even get to where 'some' data could be read. Changing skip to seek makes it so I can write the new output back to disk. I had a dd of more than a count of 5 that I could put back (in hopes of always being to undo that damage). It was strange that the changes did not always produce predictable results. Changing tmp to where I could partially read it after mounting it (read only) was done 1 way, Different changes were required for the other raid5s and combining those changes lead to more results as to the errors I would get on bootup.
I found the needed combination to be: changing all the subdisks to a good state, booting the computer (or loading geom_vinum), mounting the partitions, then unplugging ad8 leads to a disk that I can access as it was before I tampered with the 'down, but plugged in' ad8 in my attempts to consider moving from gvinum. A missing ad8 leads to not being able to mount, and an available and up ad8 leads to much corruption on the file system. Mounting and then unplugging the disk makes it happy though by my testing of filesystem corruption I saw not being present and I can play videos from disk (which can be gigs in size without any sign of any additional corruption).
Now that I could read the data, I used a `ccdconfig ccd0 64 none /dev/ad4s2 /dev/ad6s2` and a newfs across it to copy all but the swap partition by using a dump/restore. Is it just me, or does everyone else always forget that restore restores to the current directory instead of a directory added as an ignored paramater? `dump -0 -f - /dev/gvinum/usr|buffer -S 2058K -p 75|restore -r -f -` was one used command (and took about 13 hours to complete). By using ccd0 instead of ad8, I was able to keep ad8 in tact in case I needed additional attempts to access/copy the data.
Now that I had a copy of the data, I tested a fsck_ffs across /dev/gvinum/tmp and all I ended up with was one lost+found directory which took up 410K when the old one took up 2K, but there was 747MB in the tmp directory, so a lot was missing. I expected bad things to happen, which is why I try to copy/backup what I can read 'before' I try to fix a broken file system.
I should be able to newfs /dev/gvinum/ partitions and dump/restore to them from my ad*s2 partitions and follow that with a rebuildparity of ad8 to safely get things back to how they were for my previousely semi-stable but booting system right? Before I destroy/overwrite any more data, would any other information be useful to examine/debug? If ad8 had really failed, could I still have gotten back that data?
Below is the configuration. the disks go in reverse order from the device nodes numerically because I reversed them while playing with testing vinum back when I first set it up. disk1 used to be ad4 when I first set up gvinum. The configuration of `gvinum list` reflects the state of the raid5 partitions having been set to up (by manually edititng), mounting the partitions, then unplugging the drive. This is still on the boot that I made the copy to the ccd layout with.
darkstar# gvinum printconfig
# Vinum configuration of darkstar.localhost, saved at Fri Nov 16 00:17:31 2007
drive disk3 device /dev/ad4s1h
drive disk2 device /dev/ad6s1h
drive disk1 device /dev/ad8s1h
volume tmp
volume var
volume usr
volume swap
volume root
plex name tmp.p0 org raid5 8126s vol tmp
plex name var.p0 org raid5 8126s vol var
plex name usr.p0 org raid5 8126s vol usr
plex name swap.p0 org concat vol swap
plex name root.p0 org concat vol root
plex name root.p1 org concat vol root
plex name root.p2 org concat vol root
sd name tmp.p0.s2 drive disk3 len 3071628s driveoffset 387628721s plex tmp.p0 plexoffset 16252s
sd name tmp.p0.s1 drive disk2 len 3071628s driveoffset 387628721s plex tmp.p0 plexoffset 8126s
sd name tmp.p0.s0 drive disk1 len 3071628s driveoffset 387628721s plex tmp.p0 plexoffset 0s
sd name var.p0.s2 drive disk3 len 5119380s driveoffset 382508721s plex var.p0 plexoffset 16252s
sd name var.p0.s1 drive disk2 len 5119380s driveoffset 382508721s plex var.p0 plexoffset 8126s
sd name var.p0.s0 drive disk1 len 5119380s driveoffset 382508721s plex var.p0 plexoffset 0s
sd name usr.p0.s2 drive disk3 len 379882374s driveoffset 2621424s plex usr.p0 plexoffset 16252s
sd name usr.p0.s1 drive disk2 len 379882374s driveoffset 2621424s plex usr.p0 plexoffset 8126s
sd name usr.p0.s0 drive disk1 len 379882374s driveoffset 2621424s plex usr.p0 plexoffset 0s
sd name swap.p0.s2 drive disk3 len 1572583s driveoffset 265s plex swap.p0 plexoffset 3145166s
sd name swap.p0.s1 drive disk2 len 1572583s driveoffset 265s plex swap.p0 plexoffset 1572583s
sd name swap.p0.s0 drive disk1 len 1572583s driveoffset 265s plex swap.p0 plexoffset 0s
sd name root.p0.s0 drive disk1 len 1048576s driveoffset 1572848s plex root.p0 plexoffset 0s
sd name root.p1.s0 drive disk2 len 1048576s driveoffset 1572848s plex root.p1 plexoffset 0s
sd name root.p2.s0 drive disk3 len 1048576s driveoffset 1572848s plex root.p2 plexoffset 0s
darkstar# gvinum list
3 drives:
D disk3 State: up /dev/ad4s1h A: 2/190771 MB (0%)
D disk2 State: up /dev/ad6s1h A: 2/190771 MB (0%)
D disk1 State: down /dev/ad8s1h A: 2/190771 MB (0%)
5 volumes:
V tmp State: up Plexes: 1 Size: 2999 MB
V var State: up Plexes: 1 Size: 4999 MB
V usr State: up Plexes: 1 Size: 362 GB
V swap State: down Plexes: 1 Size: 2303 MB
V root State: up Plexes: 3 Size: 512 MB
7 plexes:
P tmp.p0 R5 State: degraded Subdisks: 3 Size: 2999 MB
P var.p0 R5 State: degraded Subdisks: 3 Size: 4999 MB
P usr.p0 R5 State: degraded Subdisks: 3 Size: 362 GB
P swap.p0 C State: down Subdisks: 3 Size: 2303 MB
P root.p0 C State: down Subdisks: 1 Size: 512 MB
P root.p1 C State: up Subdisks: 1 Size: 512 MB
P root.p2 C State: up Subdisks: 1 Size: 512 MB
15 subdisks:
S tmp.p0.s2 State: up D: disk3 Size: 1499 MB
S tmp.p0.s1 State: up D: disk2 Size: 1499 MB
S tmp.p0.s0 State: down D: disk1 Size: 1499 MB
S var.p0.s2 State: up D: disk3 Size: 2499 MB
S var.p0.s1 State: up D: disk2 Size: 2499 MB
S var.p0.s0 State: down D: disk1 Size: 2499 MB
S usr.p0.s2 State: up D: disk3 Size: 181 GB
S usr.p0.s1 State: up D: disk2 Size: 181 GB
S usr.p0.s0 State: down D: disk1 Size: 181 GB
S swap.p0.s2 State: up D: disk3 Size: 767 MB
S swap.p0.s1 State: up D: disk2 Size: 767 MB
S swap.p0.s0 State: down D: disk1 Size: 767 MB
S root.p0.s0 State: down D: disk1 Size: 512 MB
S root.p1.s0 State: up D: disk2 Size: 512 MB
S root.p2.s0 State: up D: disk3 Size: 512 MB
--
Ulf Lilleengen
_________________________________________________________________
Boo! Scare away worms, viruses and so much more! Try Windows Live OneCare!
http://onecare.live.com/standard/en-us/purchase/trial.aspx?s_cid=wl_hotmailnews
More information about the freebsd-geom
mailing list