iSCSI/ZFS strangeness

[Jan Bramkamp]

On 29/10/15 02:57, Michael W. Lucas wrote:
> The initiators can both access the iSCSI-based pool--not
> simultaneously, of course. But CARP, devd, and some shell scripting
> should get me a highly available pool that can withstand the demise of
> any one iSCSI server and any one initiator.
>
> The hope is that the pool would continue to work even if an iSCSI host
> shuts down. When the downed iSCSI host returns, the initiators should
> log back in and the pool auto-resilver.

I would recommend against using CARP for this because CARP is prone to 
split-brain situations and in this case they could destroy your whole 
storage pool. If the current head node fails the replacement has to 
`zpool import -f` the pool and and in the case of a split-brain 
situation both head nodes would continue writing to the iSCSI targets.

I would move the leader election to an external service like consul, 
etcd or zookeeper. This is one case where the added complexity is worth 
it. If you can't run an external service for this e.g. it would exceed 
the scope of the chapter you're writing please simplify the setup with 
more reliable hardware, good monitoring and manual failover for 
maintenance. CARP isn't designed to implement reliable (enough) master 
election for your storage cluster.

Adding iSCSI to your storage stack adds complexity and overhead. For 
setups which still fit inside a single rack SAS (with geom_multipath) is 
normally faster and cheaper. On the other hand you can't spread out SAS 
storage far enough to implement disaster tolerance should you really 
need it and it certainly is an interesting setup.