mlx5 irq

[Hans Petter Selasky]

On 2020-10-01 09:39, Michal Vančo via freebsd-net wrote:
> Hi

Hi Michal,

> I have a server with one Mellanox ConnectX-4 adapter and the following
> CPU configuration (SMT disabled):
> 
> # dmesg | grep SMP
> FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
> FreeBSD/SMP: 2 package(s) x 8 core(s) x 2 hardware threads
> FreeBSD/SMP Online: 2 package(s) x 8 core(s)
> 
> What I don't understand is the number of IRQs allocated for each mlx5_core:
> 
> # vmstat -i | grep mlx5_core
> irq320: mlx5_core0                     1          0
> irq321: mlx5_core0              18646775         84
> irq322: mlx5_core0                    21          0
> irq323: mlx5_core0                 97793          0
> irq324: mlx5_core0                 84685          0
> irq325: mlx5_core0                 89288          0
> irq326: mlx5_core0                 93564          0
> irq327: mlx5_core0                 86892          0
> irq328: mlx5_core0                 99141          0
> irq329: mlx5_core0                 86695          0
> irq330: mlx5_core0                104023          0
> irq331: mlx5_core0                 85238          0
> irq332: mlx5_core0                 88387          0
> irq333: mlx5_core0              93310221        420

^^^ it appears you have some application which is using a single TCP 
connection heavily. Then the traffic doesn't get distributed.

> irq334: mlx5_core0               1135906          5
> irq335: mlx5_core0                 85394          0
> irq336: mlx5_core0                 88361          0
> irq337: mlx5_core0                 88826          0
> irq338: mlx5_core0              17909515         81
> irq339: mlx5_core1                     1          0
> irq340: mlx5_core1              18646948         84
> irq341: mlx5_core1                    25          0
> irq342: mlx5_core1                208684          1
> irq343: mlx5_core1                 91567          0
> irq344: mlx5_core1                 88340          0
> irq345: mlx5_core1                 92597          0
> irq346: mlx5_core1                 85108          0
> irq347: mlx5_core1                 98858          0
> irq348: mlx5_core1                 88103          0
> irq349: mlx5_core1                104906          0
> irq350: mlx5_core1                 84947          0
> irq351: mlx5_core1                 99767          0
> irq352: mlx5_core1               9482571         43
> irq353: mlx5_core1               1724267          8
> irq354: mlx5_core1                 96698          0
> irq355: mlx5_core1                473324          2
> irq356: mlx5_core1                 86760          0
> irq357: mlx5_core1              11590861         52
> 
> I expected number of IRQs to be equal number of CPUS. According to
> Mellanox docs, I should be able to pin each interrupt to specific core
> to loadbalance. How can I do this in this case when number of IRQs is
> larger than number of cores? Is there any way to lower the number of
> interrupts?
> 

You can lower the number of interrupts by changing the coalescing 
sysctl's in the mce.<N>.conf tree.

dev.mce.0.conf.tx_coalesce_pkts: 32
dev.mce.0.conf.tx_coalesce_usecs: 16
dev.mce.0.conf.rx_coalesce_pkts: 32
dev.mce.0.conf.rx_coalesce_usecs: 3

For example 1024 pkts and 125 us.

And also set the queue size bigger than 1024 pkts:

dev.mce.0.conf.rx_queue_size: 1024
dev.mce.0.conf.tx_queue_size: 1024

--HPS