Would ZFS and gmirror work well together in a two-node failover cluster?

[Maurice Volaski]

I am looking to put together a two-node high-availability cluster 
where each node has identical data storage consisting of a set of 
internal data drives (separate from the boot drive). I want ZFS to 
manage the drives as a JDBOD in a RAIDZ2 configuration. Thus, if an 
individual drive misbehaves or fails, ZFS detects and handles the 
fault.

But I'm also looking to mirror this entire setup in real time to a 
second identical server.

Basically, my question is can this work well on FreeBSD while taking 
full advantage of ZFS?

Specifically, my understanding is that the only way to handle the 
real time mirror is with gmirror and ggated, but it's not clear how 
gmirror would interact with ZFS.

I am assuming that gmirror operates only on individual drives, so if 
I had a set of 24 drives on each server, there would be 24 mirrored 
drive pairs.

One concern I have is that this setup could run into trouble with 
gmirror's potentially sabotaging ZFS's RAIDZ2. For example, when a 
drive starts failing, won't gmirror see it before ZFS does and take 
the unfavorable action of substituting the corresponding drive in the 
failover server in subsequent I/O, leaving ZFS's RAIDZ2 out of the 
loop?

This is just one particular scenario, but in general, it's not 
entirely clear that it's possible to have fine-grained control of 
when, how much and in what direction gmirror manages synchronization 
among drive pairs.

-Maurice
-- 

Maurice Volaski, mvolaski at aecom.yu.edu
Computing Support, Rose F. Kennedy Center
Albert Einstein College of Medicine of Yeshiva University