Uneven load on drives in ZFS RAIDZ1

[Garrett Cooper]

On Dec 19, 2011, at 12:54 PM, Stefan Esser wrote:

> Am 19.12.2011 18:05, schrieb Garrett Cooper:
>> On Mon, Dec 19, 2011 at 6:22 AM, Stefan Esser <se at freebsd.org> wrote:
>>> Hi ZFS users,
>>> 
>>> for quite some time I have observed an uneven distribution of load
>>> between drives in a 4 * 2TB RAIDZ1 pool. The following is an excerpt of
>>> a longer log of 10 second averages logged with gstat:
>>> 
>>> dT: 10.001s  w: 10.000s  filter: ^a?da?.$
>>> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>>>   0    130    106   4134    4.5     23   1033    5.2   48.8| ada0
>>>   0    131    111   3784    4.2     19   1007    4.0   47.6| ada1
>>>   0     90     66   2219    4.5     24   1031    5.1   31.7| ada2
>>>   1     81     58   2007    4.6     22   1023    2.3   28.1| ada3
>>> 
>>> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>>>   1    132    104   4036    4.2     27   1129    5.3   45.2| ada0
>>>   0    129    103   3679    4.5     26   1115    6.8   47.6| ada1
>>>   1     91     61   2133    4.6     30   1129    1.9   29.6| ada2
>>>   0     81     56   1985    4.8     24   1102    6.0   29.4| ada3
>>> 
>>> L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
>>>   1    148    108   4084    5.3     39   2511    7.2   55.5| ada0
>>>   1    141    104   3693    5.1     36   2505   10.4   54.4| ada1
>>>   1    102     62   2112    5.6     39   2508    5.5   35.4| ada2
>>>   0     99     60   2064    6.0     39   2483    3.7   36.1| ada3
>> 
>> This suggests (note that I said suggests) that there might be a slight
>> difference in the data path speeds or physical media as someone else
>> suggested; look at zpool iostat -v <interval> though before making a
>> firm statement as to whether or not a drive is truly not performing to
>> your assumed spec. gstat and zpool iostat -v suggest performance
>> though -- they aren't the end-all-be-all for determining drive
>> performance.
> 
> I doubt there is a difference in the data path speeds, since all drives
> are connected to the SATA II ports of an Intel H67 chip.
> 
> The drives seem to perform equally well, just with a ratio of read
> requests of 30% / 30% / 20% / 20% for ada0 .. ada3. But neither queue
> length nor command latencies indicate a problem or differences in the
> drives. It seems that a different number of commands is scheduled for 2
> of the 4 drives, compared to the other 2, and that scheduling should be
> part of the ZFS code. I'm quite convinced, that neither the drives nor
> the other hardware plays a role, but I'll follow the suggestion to swap
> drives between controller ports and to observe whether the increased
> read load moves with the drives (indicating something on disk causes the
> anomaly) or stays with the SATA ports (indicating that lower numbered
> ports see higher load).
> 
>> If the latency numbers were high enough, I would suggest dd'ing out to
>> the individual drives (i.e. remove the drive from the RAIDZ) to see if
>> there's a noticeable discrepancy, as this can indicate a bad cable,
>> backplane, or drive; from there I would start doing the physical swap
>> routine and see if the issue moves with the drive or stays static with
>> the controller channel and/or chassis slot.
> 
> I do not expect a hardware problem, since command latencies are very
> similar over all drives, despite the higher read load on some of them.
> These are more busy by exactly the factor to be expected by only the
> higher command rate.
> 
> But it seems that others do not observe the asymmetric distribution of
> requests, which makes me wonder whether I happen to have meta data
> arranged in such a way that it is always read from ada0 or ada1, but not
> (or rarely) from ada2 or ada3. That could explain it, including the fact
> that raidz1 over other numbers of drives 8e.g. 3 or 6) apparently show a
> much more symmetric distribution of read requests.

Basic question: does one set of drives vibrate differently than the other set?
-Garrett