ZFS v13 running vicious replacement loop.

[Jørgen Dovregubbe Næss]

Hi.

Im running freeNAS v.0.7.2 with zfs v3 and zpool v13.

I was trying to upgrade my raidz1 from 1 TB drives to 2TB drives, replacing
1 drive at a time.

here's the current layout:

freenas:~# zpool status
  pool: guitar
 state: DEGRADED
 scrub: resilver completed after 0h0m with 0 errors on Sat Jul 16 17:19:48
2011
config:

        NAME                       STATE     READ WRITE CKSUM
        guitar                     DEGRADED     0     0     0
          raidz1                   DEGRADED     0     0     0
            replacing              DEGRADED     0    25     4
              ad4.nop              ONLINE       0     0     0  88K
resilvered
              9203819577564534483  FAULTED      0     0     0  was
/dev/ad4.nop
            ad6.nop                ONLINE       0     0     0  96K
resilvered
            ad8.nop                ONLINE       0     0     0  100K
resilvered
            ad10.nop               ONLINE       0     0     0  96K
resilvered
            ad16.nop               ONLINE       0     0     0  124K
resilvered
            ad18.nop               ONLINE       0     0     0  116K
resilvered
        spares
          da1                      AVAIL

errors: No known data errors

Without taking ad4.nop offline, i turned the machine off, replaced the
drive, and did a "zpool replace guitar ad4.nop ad4"

It started replacing the drive, but got I/o write and read errors, so it
restarted (happened like 30 times per minute).

I have tried everything.

zpool replace guitar ad4.nop /dev/da1 (can't replace because already being
replaced)

zpool offline guitar ad4.nop (no valid replicas)

zpool offline guitar 9203819577564534483 (no valid replicas)

zpool clear guitar ad4.nop

zpool export / import (thus the 9203819577564534483, it said ad4.nop before)

The spare was added later to try and put that in instead of ad4.nop, without
luck.
The box only have 6 sata ports, so I am unable to connect the drive i was
trying to replace ad4.nop with.

I do have another computer that i could use as a temp server, but that one
also has 6 sata ports.
The next raidz1 will be with only 5 drives, and 1 spare ( so i dont have to
meet this problem again)

The problem now is, that i can't access the files that are on the NAS, thus
being unable to do a backup of it. ( I can transfer like a coulple of GB,
then the whole box gets unreachable (web, ssh ftp).

Any Idea how to solve this?

Best Regards
Jørgen