Adding to a zpool -- different redundancies and risks

[David Christensen]

On 2019-12-11 09:39, Norman Gray wrote:
> 
> Greetings.
> 
> I'd like to add a new VDEV to a pool, and I'm being warned (slightly to
> my surprise) that there's a 'mismatched replication level'.  I'm trying
> to get a sense of how much of a risk I'd be running by forcing this with
> -f.
> 
> Context:
> 
>     * I currently have two raidz2 VDEVs composed of nine 5.5TB disks
> (thus 2 x ~40TB available)
>     * I'd like to add another raidz2 VDEV composed of six 12TB disks
> (thus adding a VDEV of ~48TB, roughly the same size as the other two) --
> this is what prompts the warning about replication level
>     * The storage is a local mirror which it would be very annoying to
> lose, but it's not holding unique copies of anything
>     * I don't like using -f options unless I'm pretty damn confident I
> know what's happening
> 
> (I wouldn't set this up in quite this way from scratch, but this is an
> old-ish server, and a small budget windfall has allowed me to max out
> the remaining available slots with new disks).
> 
> I can appreciate that the ideal planned setup would, in principle, be to
> have all the VDEVs be symmetrical, in terms of size and number of disks.
>    Is a VDEV mix merely 'not ideal', or 'not great but you'll be fine',
> or Bad?
> 
> My mental model of what's going on suggests that, since the pool simply
> stripes across the VDEVs, it doesn't have to care how the VDEVs
> themselves are structured, so that a 9x5.5 raidz2 and a 6x12 raidz2
> would be roughly equally used, and I can't see why there would be a
> performance or a utilisation difference between the two (but I still
> count myself as a ZFS tyro).
> 
> I can see that there would be a reliability issue if the various VDEVs
> were different sizes of mirrored ones -- this would create different
> amounts of resilience, and so the warning makes sense in a 'are you
> sure?' way.  If the VDEVs were different sizes and the pool was
> mirroring over them, then there would obviously be a utilisation issue.
> 
> Though both of [1] and [2] illustrate only mixing VDEVs of the same
> type, [2] says merely that 'When using RAIDZ vdevs, it is also a good
> idea to keep them at the same width and of the same type.' and
> illustrates a 4-wide plus 8-wide raidz2 as 'not horrible'.  The forum
> post at [3] asks essentially the same question as this email, but
> receives a rather oblique answer.  The question at [4] gets a confident
> answer which I don't _think_ makes complete sense.
> 
> Since [1] and [2] are both more authoritative and match my own
> understanding, I'm inclined to believe that adding this new VDEV would
> be less than perfect, but reasonable.  Am I deceiving myself?
> 
> Thanks for any advice you can offer,
> 
> Norman
> 
> 
> [1]
> https://forums.freenas.org/index.php?threads/slideshow-explaining-vdev-zpool-zil-and-l2arc-for-noobs.7775/
> [2]
> https://www.ixsystems.com/community/resources/introduction-to-zfs.111/
> [3]
> https://forums.freebsd.org/threads/zfs-mismatched-repli-levels.28226/
> [4] https://serverfault.com/questions/522782/zfs-with-unsymmetric-vdevs
> 

Please post:

1   The 'zpool create ...' command you used to create the existing pool.

2.  The output of 'zpool status' for the existing pool.

3.  The output of 'zpool list' for the existing pool.

4.  The 'zpool add ...' command you are contemplating.


So, you have 24 drives in a 24 drive cage?


What are your space and performance goals?


What are your sustainability goals as drives and/or VDEV's fail?


David