ZfS & GEOM with many odd drive sizes

[Doug Rabson]

On Wed, 2007-07-25 at 10:23 +0100, Mark Powell wrote:
> On Sat, 21 Jul 2007, Pawel Jakub Dawidek wrote:
> 
> Thanks for your reply.
> 
> > Be sure to turn off debugging, ie. remove WITNESS, INVARIANTS and
> > INVARIANT_SUPPORT options from your kernel configuration.
> > Other than that, ZFS may just be more CPU hungry...
> 
>    I have. Makes little difference. Think the idea of using an Athlon XP 
> for ZFS has turned out to be a bridge too far. The new 65nm Athlon 64 x2 
> are very cheap now. Time for an upgrade.
>    You said that replacing one device with another is not a problem. Just 
> to be clear on this as it's a key factor in me going with this solution. I 
> hope this isn't too naive a question, but the answer will be here for 
> others :)
>    Suppose instead of gconcat I used gstripe on the 250+200 combinations:
> 
> i.e. (slice 1 on all drives is reserved for ufs gmirror of /boot and 
> block device swap)
> 
> gs0 ad0s2 ad1s2
> gs1 ad2s2 ad3s2
> gs2 ad4s2 ad5s2
> 
> I use these gstripes and the single 400GB drive to construct the zpool:
> 
> zpool create tank raidz /dev/mirror/gs0 /dev/mirror/gs1 /dev/mirror/gs2 ad6s2
> 
> If for example ad3 fails and thus gs1 fails, how is this replaced in the 
> zpool? e.g. suppose I replace both ad2 and ad3 with a new 500GB drive as 
> ad2. Is fixing this as simple as:
> 
> zpool replace tank /dev/mirror/gs1 ad2s2
> 
> Many thanks.

I'm not really sure why you are using gmirror, gconcat or gstripe at
all. Surely it would be easier to let ZFS manage the mirroring and
concatentation. If you do that, ZFS can use its checksums to continually
monitor the two sides of your mirrors for consistency and will be able
to notice as early as possible when one of the drives goes flakey. For
concats, ZFS will also spread redundant copies of metadata (and regular
data if you use 'zfs set copies=<N>') across the disks in the compat. If
you have to replace one half of a mirror, ZFS has enough information to
know exactly which blocks needs to be copied to the new drive which can
make recovery much quicker.