mlx5 interrupts

[Slawa Olhovchenkov]

On Mon, Jul 20, 2020 at 02:31:48PM +0300, Konstantin Belousov wrote:

> On Mon, Jul 20, 2020 at 02:04:26PM +0300, Slawa Olhovchenkov wrote:
> > On Mon, Jul 20, 2020 at 01:43:39PM +0300, Konstantin Belousov wrote:
> > 
> > > On Sun, Jul 19, 2020 at 07:12:07PM +0300, Slawa Olhovchenkov wrote:
> > > > Can anybody explain what purpose of unnamed interrupts of Mellanox
> > > > ConnectX-5 cards?
> > > > 
> > > > I am see 19 interrupts per card. I am mean last 16 is RX queue.
> > > It is more correct to describe them as completion queue interrupts, they
> > > serve both rx and tx workqueues.
> > > 
> > > > What about first 3?
> > > One is for firmware memory requests notifications for UMA.
> > > Two others are firmware command completion and async events (like PHY
> > > module events or hw state notifications).
> > > 
> > > > Also I am see very high rate for irq287/irq306 -- is this good?
> > > You did not demostrated what you mean by 'very high rate'.
> > > 20 req/sec does not count as 'high', and this is the highest I see
> > > in your message.
> > 
> > Currently this is idle server and totaly interrupts count for firmware
> > command completion is 50x more compared to completion queue
> > interrupts. I am afraid to flooded at prodation load.
> Did you reflashed firmware to the latest supported release ?
> If not, do that.

Don't release number in logs, only show by melanox utilites?

> Why are you sure that this is command completion and not events ?

You words:

One is for firmware memory requests notifications for UMA
firmware command completion
async events

-- I mean you named in order.

Unfortunately all interrupts named identicaly

> Set sysctl dev.mce.0.conf.debug_stats=1 then look at the output
> of sysctl dev.mce.0.hw_ctx_debug.
> 

dev.mce.0.hw_ctx_debug: pages irq 286
command irq 287
async irq 288
channel 0 rq 353 cq 1025 irq 289
channel 0 tc 0 sq 352 cq 1024 irq 289


> > Also, completion queue interrupts may be pined to dedicaded CPU, what
> > about command completion?
> Command completion is normal MSI-X interrupt, but I think that the official
> recommendation is to not fiddle with it.  Commands are not used for normal
> tx/rx path, they only provide management and statistics.
> 
> > 
> > > > 
> > > > # vmstat -i | grep -e ^int -e mlx
> > > > interrupt                          total       rate
> > > > irq286: mlx5_core0                     1          0
> > > > irq287: mlx5_core0               5135992         20
> > > > irq288: mlx5_core0                     1          0
> > > > irq289: mlx5_core0                 76408          0
> > > > irq290: mlx5_core0                 43054          0
> > > > irq291: mlx5_core0                 93826          0
> > > > irq292: mlx5_core0                 39457          0
> > > > irq293: mlx5_core0                 36141          0
> > > > irq294: mlx5_core0                 65526          0
> > > > irq295: mlx5_core0                 53399          0
> > > > irq296: mlx5_core0                120885          0
> > > > irq297: mlx5_core0                140690          1
> > > > irq298: mlx5_core0                193578          1
> > > > irq299: mlx5_core0                178332          1
> > > > irq300: mlx5_core0                 75334          0
> > > > irq301: mlx5_core0                207118          1
> > > > irq302: mlx5_core0                108803          0
> > > > irq303: mlx5_core0                 24356          0
> > > > irq304: mlx5_core0                 26713          0
> > > > irq305: mlx5_core1                     1          0
> > > > irq306: mlx5_core1               5136296         20
> > > > irq307: mlx5_core1                     1          0
> > > > irq308: mlx5_core1               3634544         14
> > > > irq309: mlx5_core1                 22860          0
> > > > irq310: mlx5_core1                564441          2
> > > > irq311: mlx5_core1                 30503          0
> > > > irq312: mlx5_core1                115549          0
> > > > irq313: mlx5_core1                 49815          0
> > > > irq314: mlx5_core1                 10272          0
> > > > irq315: mlx5_core1                 85875          0
> > > > irq316: mlx5_core1                134251          1
> > > > irq317: mlx5_core1                 25151          0
> > > > irq318: mlx5_core1                 73376          0
> > > > irq319: mlx5_core1                  5879          0
> > > > irq320: mlx5_core1                 39515          0
> > > > irq321: mlx5_core1                  5390          0
> > > > irq322: mlx5_core1                 22726          0
> > > > irq323: mlx5_core1                 60408          0