ZFS stalled after some mirror disks were lost

[Steven Hartland]

On 03/10/2017 08:31, Ben RUBSON wrote:
>> On 03 Oct 2017, at 09:25, Steven Hartland <steven at multiplay.co.uk> wrote:
>>
>> On 03/10/2017 07:12, Andriy Gapon wrote:
>>> On 02/10/2017 21:12, Ben RUBSON wrote:
>>>
>>>> A sustained read throughput of 180 MB/s, 45 MB/s on each iscsi disk
>>>> according to "zpool iostat", nothing on local disks (strange but I
>>>> noticed that IOs always prefer iscsi disks to local disks).
>>>>
>>> Are your local disks SSD or HDD?
>>> Could it be that iSCSI disks appear to be faster than the local disks to the
>>> smart ZFS mirror code?
>>>
>>> Steve, what do you think?
>>>
>> Yes that quite possible, the mirror balancing uses the queue depth + rotating bias to determine the load of the disk so if your iSCSI host is processing well and / or is reporting non-rotating vs rotating for the local disks it could well be the mirror is preferring reads from the the less loaded iSCSI devices.
> Note that local & iscsi disks are _exactly_ the same (same model number, same SAS adapter...).
> So iSCSI ones should be a little bit slower due to network latency (even if it's very low in my case).
> Once production back, after having analysed the main issue of this thread, I should then
> try to find whether or not iSCSI disks are seen as rotating disks.
>
> Thanks for the hint !
Hmm, the output from gstat -dp on a loaded machine would be interesting 
to see too.

     Regards
     Steve