vinum question: ide chain crashed, one or two drives dead?

[Joseph Kerian]

I was away for a long weekend when I recieved an annoyed phone call
regarding a site that I manage. While it was not terribly difficult to
bring the website back online, I lost either a controller card or an
IDE cable, and this produced a string of errors that eventually led to
a kernel panic. I would very much like to be able to recover some of
this data, and am curious if I am able to.

Running FreeBSD nene 5.3-RELEASE-p5 with a custom kernel to include
IPFW options.

The following was my "var/log/messages" for the relevent times:
Jun 20 04:30:44 nene kernel: ad2: WARNING - removed from configuration
Jun 20 04:30:44 nene kernel: ad3: WARNING - removed from configuration
Jun 20 04:30:44 nene kernel: ata1-master: FAILURE - unknown CMD (0xb0) timed out
Jun 20 04:30:44 nene kernel: vinum: ideraid.p0.s0 is crashed by force
Jun 20 04:30:44 nene kernel: vinum: ideraid.p0 is degraded
Jun 20 04:30:44 nene kernel: fatal:ideraid.p0.s0 read error, block
231758025 for 16384 bytes
Jun 20 04:30:44 nene kernel: ideraid.p0.s0: user buffer block
1158788032 for 16384 bytes
Jun 20 04:30:44 nene kernel: dua: fatal drive I/O error, block
231758025 for 16384 bytes
Jun 20 04:30:44 nene kernel: vinum: drive dua is down
Jun 20 04:30:44 nene kernel: vinum: ideraid.p0.s0 is stale by force
Jun 20 04:30:44 nene kernel: fatal :ideraid.p0.s0 write error, block
231758025 for 16384 bytes
Jun 20 04:30:44 nene kernel: ideraid.p0.s0: user buffer block
1158788032 for 16384 bytes
Jun 20 04:30:44 nene kernel: dua: fatal drive I/O error, block
231758025 for 16384 bytes
Jun 20 04:30:44 nene smartd[604]: Device: /dev/ad2, failed to read
SMART Attribute Data
Jun 20 04:30:44 nene kernel: vinum: Can't write config to /dev/ad3a, error 6
Jun 20 04:30:44 nene kernel: vinum: drive eva34 is down
Jun 20 04:30:44 nene kernel: vinum: ideraid.p0.s1 is crashed
Jun 20 04:30:44 nene kernel: vinum: ideraid.p0 is corrupt
Jun 21 03:01:08 nene kernel: fatal:ideraid.p0.s1 read error, block
301684457 for 16384 bytes
Jun 21 03:01:08 nene kernel: ideraid.p0.s1: user buffer block
1508419040 for 16384 bytes
Jun 21 03:01:08 nene kernel: eva34: fatal drive I/O error, block
301684457 for 16384 bytes
Jun 21 03:01:08 nene kernel: fatal:ideraid.p0.s1 read error, block
301684457 for 16384 bytes
Jun 21 03:01:08 nene kernel: ideraid.p0.s1: user buffer block
1508419040 for 16384 bytes
Jun 21 03:01:08 nene kernel: eva34: fatal drive I/O error, block
301684457 for 16384 bytes
Jun 21 03:01:09 nene kernel: 4 bytes
Jun 21 03:01:09 nene kernel: fatal:ideraid.p0.s1 read error, block
301684457 for 16384 bytes
Jun 21 03:01:09 nene kernel: ideraid.p0.s1: user buffer block
1508419040 for 16384 bytes
Jun 21 03:01:09 nene kernel: eva34: fatal drive I/O error, block
301684457 for 16384 bytes
(approximately 3000 repeats of those 3 lines later)
Jun 21 03:01:37 nene kernel: vinum: ideraid.p0.s1 is stale by force
Jun 21 03:01:37 nene kernel: vinum: ideraid.p0.s0 is crashed by force
Jun 21 03:01:37 nene kernel: fatal:ideraid.p0.s0 read error, block
177262153 for 16384 bytes
Jun 21 03:01:37 nene kernel: ideraid.p0.s0: user buffer block
886309184 for 16384 bytes
Jun 21 03:01:37 nene kernel: dua: fatal drive I/O error, block
177262153 for 16384 bytes
Jun 21 03:01:37 nene kernel: fatal:ideraid.p0.s0 read error, block
177563337 for 16384 bytes
Jun 21 03:01:37 nene kernel: ideraid.p0.s0: user buffer block
887814592 for 16384 bytes
Jun 21 03:01:37 nene kernel: dua: fatal drive I/O error, block
177563337 for 16384 bytes
Jun 21 03:01:37 nene kernel: fatal:ideraid.p0.s0 read error, block
178466889 for 16384 bytes
Jun 21 03:01:37 nene kernel: ideraid.p0.s0: user buffer block
892330816 for 16384 bytes
Jun 21 03:01:37 nene kernel: dua: fatal drive I/O error, block
178466889 for 16384 bytes
Jun 21 03:01:37 nene kernel: vinum: ideraid.p0.s0 is stale by force

vinum list: (after using "start")
6 drives:
D gva250                State: up   /dev/ad7a   A: 0/238475 MB (0%)
D eva200                State: up   /dev/ad6a   A: 0/190782 MB (0%)
D gva200                State: up   /dev/ad5a   A: 0/190782 MB (0%)
D eva250                State: up   /dev/ad4a   A: 0/238475 MB (0%)
D eva34                 State: up   /dev/ad3a   A: 0/190782 MB (0%)
D dua                   State: up   /dev/ad2a   A: 0/190782 MB (0%)

2 volumes:
V ideraid               State: up   Plexes:       1   Size:        931 GB
V mirror2               State: up   Plexes:       2   Size:         46 GB

3 plexes:
P ideraid.p0         R5 State: corrupt Subdisks:     6   Size:        931 GB
P mirror2.p0          C State: up   Subdisks:     1   Size:         46 GB
P mirror2.p1          C State: up   Subdisks:     1   Size:         46 GB

8 subdisks:
S ideraid.p0.s0         State: stale   D: dua          Size:        186 GB
S ideraid.p0.s1         State: crashed D: eva34        Size:        186 GB
S ideraid.p0.s2         State: up   D: eva250       Size:        186 GB
S ideraid.p0.s3         State: up   D: gva200       Size:        186 GB
S ideraid.p0.s4         State: up   D: eva200       Size:        186 GB
S ideraid.p0.s5         State: up   D: gva250       Size:        186 GB
S mirror2.p0.s0         State: up   D: gva250       Size:         46 GB
S mirror2.p1.s0         State: up   D: eva250       Size:         46 GB

vinum_history contains nothing of interest, a long history of "l"
(list), start, stop, and a few helps. I'm not entirely sure why it
doesn't contain the commands of initially creating the array. But the
array appears in that vinum list as it was configured, although it
would be nice if the ideraid array functioned.

Any help in this would be greatly appreciated.

--Joe Kerian