Adding to a zpool -- different redundancies and risks

[Norman Gray]

Greetings.

I'd like to add a new VDEV to a pool, and I'm being warned (slightly to 
my surprise) that there's a 'mismatched replication level'.  I'm trying 
to get a sense of how much of a risk I'd be running by forcing this with 
-f.

Context:

   * I currently have two raidz2 VDEVs composed of nine 5.5TB disks 
(thus 2 x ~40TB available)
   * I'd like to add another raidz2 VDEV composed of six 12TB disks 
(thus adding a VDEV of ~48TB, roughly the same size as the other two) -- 
this is what prompts the warning about replication level
   * The storage is a local mirror which it would be very annoying to 
lose, but it's not holding unique copies of anything
   * I don't like using -f options unless I'm pretty damn confident I 
know what's happening

(I wouldn't set this up in quite this way from scratch, but this is an 
old-ish server, and a small budget windfall has allowed me to max out 
the remaining available slots with new disks).

I can appreciate that the ideal planned setup would, in principle, be to 
have all the VDEVs be symmetrical, in terms of size and number of disks. 
  Is a VDEV mix merely 'not ideal', or 'not great but you'll be fine', 
or Bad?

My mental model of what's going on suggests that, since the pool simply 
stripes across the VDEVs, it doesn't have to care how the VDEVs 
themselves are structured, so that a 9x5.5 raidz2 and a 6x12 raidz2 
would be roughly equally used, and I can't see why there would be a 
performance or a utilisation difference between the two (but I still 
count myself as a ZFS tyro).

I can see that there would be a reliability issue if the various VDEVs 
were different sizes of mirrored ones -- this would create different 
amounts of resilience, and so the warning makes sense in a 'are you 
sure?' way.  If the VDEVs were different sizes and the pool was 
mirroring over them, then there would obviously be a utilisation issue.

Though both of [1] and [2] illustrate only mixing VDEVs of the same 
type, [2] says merely that 'When using RAIDZ vdevs, it is also a good 
idea to keep them at the same width and of the same type.' and 
illustrates a 4-wide plus 8-wide raidz2 as 'not horrible'.  The forum 
post at [3] asks essentially the same question as this email, but 
receives a rather oblique answer.  The question at [4] gets a confident 
answer which I don't _think_ makes complete sense.

Since [1] and [2] are both more authoritative and match my own 
understanding, I'm inclined to believe that adding this new VDEV would 
be less than perfect, but reasonable.  Am I deceiving myself?

Thanks for any advice you can offer,

Norman


[1] 
https://forums.freenas.org/index.php?threads/slideshow-explaining-vdev-zpool-zil-and-l2arc-for-noobs.7775/
[2] 
https://www.ixsystems.com/community/resources/introduction-to-zfs.111/
[3] 
https://forums.freebsd.org/threads/zfs-mismatched-repli-levels.28226/
[4] https://serverfault.com/questions/522782/zfs-with-unsymmetric-vdevs

-- 
Norman Gray  :  https://nxg.me.uk
SUPA School of Physics and Astronomy, University of Glasgow, UK