ZFS stalled after some mirror disks were lost

[Ben RUBSON]

> On 02 Oct 2017, at 20:44, Adam Vande More <amvandemore at gmail.com> wrote:
> 
>> On Mon, Oct 2, 2017 at 1:12 PM, Ben RUBSON <ben.rubson at gmail.com> wrote:
>> Hi,
>> 
>> On a FreeBSD 11 server, the following online/healthy zpool :
>> 
>> home
>>   mirror-0
>>     label/local1
>>     label/local2
>>     label/iscsi1
>>     label/iscsi2
>>   mirror-1
>>     label/local3
>>     label/local4
>>     label/iscsi3
>>     label/iscsi4
>> cache
>>   label/local5
>>   label/local6
>> 
>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk
>> according to "zpool iostat", nothing on local disks (strange but I
>> noticed that IOs always prefer iscsi disks to local disks).
>> No write IOs.
>> 
>> Let's disconnect all iSCSI disks :
>> iscsictl -Ra
>> 
>> Expected behavior :
>> IO activity flawlessly continue on local disks.
> 
> Perhaps I'm misunderstanding your setup, but my expected behavior would be exactly what you see. 

Unfortunately, what I see is the following quoted :

>> What happened :
>> All IOs stalled, server only answers to IOs are made to its zroot pool.
>> All commands related to the iSCSI disks (iscsictl), or to ZFS (zfs/zpool),
>> don't return.

(and I would have expected the IO activity to flawlessly continue)