iSCSI failing, MLX rx_ring errors ?

[Ben RUBSON]

> On 04 Jan 2017, at 14:47, Ben RUBSON <ben.rubson at gmail.com> wrote:
> 
>> On 03 Jan 2017, at 07:27, Meny Yossefi <menyy at mellanox.com> wrote:
>> 
>>> From: owner-freebsd-net at freebsd.orgOn Behalf OfBen RUBSON
>>> Sent: Monday, January 2, 2017 11:09:15 AM (UTC+00:00) Monrovia, Reykjavik
>>> To: freebsd-net at freebsd.org
>>> Cc: Meny Yossefi; Yuval Bason; Hans Petter Selasky
>>> Subject: Re: iSCSI failing, MLX rx_ring errors ?
>>> 
>>> Hi Meny,
>>> 
>>> Thank you very much for your feedback.
>>> 
>>> I think you are right, this could be a mbufs issue.
>>> Here are some more numbers :
>>> 
>>> # vmstat -z | grep -v "0,   0$"
>>> ITEM                   SIZE   LIMIT     USED     FREE         REQ      FAIL SLEEP
>>> 4 Bucket:                32,      0,    2673,   28327,   88449799,    17317, 0
>>> 8 Bucket:                64,      0,     449,   15609,   13926386,     4871, 0
>>> 12 Bucket:               96,      0,     335,    5323,   10293892,   142872, 0
>>> 16 Bucket:              128,      0,     533,    6070,    7618615,   472647, 0
>>> 32 Bucket:              256,      0,    8317,   22133,   36020376,   563479, 0
>>> 64 Bucket:              512,      0,    1238,    3298,   20138111, 11430742, 0
>>> 128 Bucket:            1024,      0,    1865,    2963,   21162182,   158752, 0
>>> 256 Bucket:            2048,      0,    1626,     450,   80253784,  4890164, 0
>>> mbuf_jumbo_9k:         9216, 603712,   16400,    8744, 4128521064,     2661, 0
>> 
>> Have you tried increasing the mbufs limit? 
>> (sysctl) kern.ipc.nmbufs (Maximum number of mbufs allowed)
> 
> Thank you for your suggestion Meny.
> No I did not try this yet.
> However, from the numbers above (and below), I think I should increase kern.ipc.nmbjumbo9 instead ?

So, to give some news, increasing kern.ipc.nmbjumbo9 helped a lot.
Just a very little issue (compared to the others before) over the last 3 weeks.

> # vmstat -z | grep -E "ITEM|mbuf"
> ITEM              SIZE     LIMIT   USED   FREE          REQ  FAIL SLEEP
> mbuf_packet:       256, 26080395, 16400, 10418,   572292683,    0,   0
> mbuf:              256, 26080395, 16402, 40525, 20955366061,    0,   0
> mbuf_cluster:     2048,  4075060, 26818,   148,      907005,    0,   0
> mbuf_jumbo_page:  4096,  2037529,     0, 34262,  5194563127,    0,   0
> mbuf_jumbo_9k:    9216,   603712, 16400, 12867,  4362104082, 2676,   0
> mbuf_jumbo_16k:  16384,   339588,     0,     0,           0,    0,   0
> 
> # sysctl kern.ipc | grep mb
> kern.ipc.nmbufs:      26080380
> kern.ipc.nmbclusters:  4075058
> kern.ipc.nmbjumbop:    2037529
> kern.ipc.nmbjumbo9:    1811136
> kern.ipc.nmbjumbo16:   1358352
> kern.ipc.maxmbufmem: 33382879232
> 
> // note that I don't understand the difference between vmstat and sysctl nmbjumbo9 / nmbjumbo16 values.
> // the first one is /3 (1811136/3=603712) the second one is /4 (1358352/4=339588). Strange.

Any idea regarding this difference ?

In addition :
# sysctl kern.ipc.nmbjumbo9
kern.ipc.nmbjumbo9: 1811136
# sysctl kern.ipc.nmbjumbo9=2037529
kern.ipc.nmbjumbo9: 1811136 -> 6112587

Really strange.

Thank you,

Ben