mlx5 irq

[Michal Vančo]

On 01/10/2020 10:10, Hans Petter Selasky wrote:

> On 2020-10-01 09:39, Michal Vančo via freebsd-net wrote:
>> Hi
>
> Hi Michal,
>
Thank you for your quick reply.

>> I have a server with one Mellanox ConnectX-4 adapter and the following
>> CPU configuration (SMT disabled):
>>
>> # dmesg | grep SMP
>> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
>> FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 hardware threads
>> FreeBSD/SMP Online: 2 package(s) x 8 core(s)
>>
>> What I don't understand is the number of IRQs allocated for each
>> mlx5_core:
>>
>> # vmstat -i | grep mlx5_core
>> irq320: mlx5_core0                     1          0
>> irq321: mlx5_core0              18646775         84
>> irq322: mlx5_core0                    21          0
>> irq323: mlx5_core0                 97793          0
>> irq324: mlx5_core0                 84685          0
>> irq325: mlx5_core0                 89288          0
>> irq326: mlx5_core0                 93564          0
>> irq327: mlx5_core0                 86892          0
>> irq328: mlx5_core0                 99141          0
>> irq329: mlx5_core0                 86695          0
>> irq330: mlx5_core0                104023          0
>> irq331: mlx5_core0                 85238          0
>> irq332: mlx5_core0                 88387          0
>> irq333: mlx5_core0              93310221        420
>
> ^^^ it appears you have some application which is using a single TCP
> connection heavily. Then the traffic doesn't get distributed.
>
This seems correct. This machine is a NFS server with only two clients.
>> irq334: mlx5_core0               1135906          5
>> irq335: mlx5_core0                 85394          0
>> irq336: mlx5_core0                 88361          0
>> irq337: mlx5_core0                 88826          0
>> irq338: mlx5_core0              17909515         81
>> irq339: mlx5_core1                     1          0
>> irq340: mlx5_core1              18646948         84
>> irq341: mlx5_core1                    25          0
>> irq342: mlx5_core1                208684          1
>> irq343: mlx5_core1                 91567          0
>> irq344: mlx5_core1                 88340          0
>> irq345: mlx5_core1                 92597          0
>> irq346: mlx5_core1                 85108          0
>> irq347: mlx5_core1                 98858          0
>> irq348: mlx5_core1                 88103          0
>> irq349: mlx5_core1                104906          0
>> irq350: mlx5_core1                 84947          0
>> irq351: mlx5_core1                 99767          0
>> irq352: mlx5_core1               9482571         43
>> irq353: mlx5_core1               1724267          8
>> irq354: mlx5_core1                 96698          0
>> irq355: mlx5_core1                473324          2
>> irq356: mlx5_core1                 86760          0
>> irq357: mlx5_core1              11590861         52
>>
>> I expected number of IRQs to be equal number of CPUS. According to
>> Mellanox docs, I should be able to pin each interrupt to specific core
>> to loadbalance. How can I do this in this case when number of IRQs is
>> larger than number of cores? Is there any way to lower the number of
>> interrupts?
>>
>
> You can lower the number of interrupts by changing the coalescing
> sysctl's in the mce.<N>.conf tree.
>
> dev.mce.0.conf.tx_coalesce_pkts: 32
> dev.mce.0.conf.tx_coalesce_usecs: 16
> dev.mce.0.conf.rx_coalesce_pkts: 32
> dev.mce.0.conf.rx_coalesce_usecs: 3
>
> For example 1024 pkts and 125 us.
>
> And also set the queue size bigger than 1024 pkts:
>
> dev.mce.0.conf.rx_queue_size: 1024
> dev.mce.0.conf.tx_queue_size: 1024
>
But why is the actual number of IRQ lines bigger than number of CPU cores?