mlx5 interrupts

[Konstantin Belousov]

On Mon, Jul 20, 2020 at 02:04:26PM +0300, Slawa Olhovchenkov wrote:
> On Mon, Jul 20, 2020 at 01:43:39PM +0300, Konstantin Belousov wrote:
> 
> > On Sun, Jul 19, 2020 at 07:12:07PM +0300, Slawa Olhovchenkov wrote:
> > > Can anybody explain what purpose of unnamed interrupts of Mellanox
> > > ConnectX-5 cards?
> > > 
> > > I am see 19 interrupts per card. I am mean last 16 is RX queue.
> > It is more correct to describe them as completion queue interrupts, they
> > serve both rx and tx workqueues.
> > 
> > > What about first 3?
> > One is for firmware memory requests notifications for UMA.
> > Two others are firmware command completion and async events (like PHY
> > module events or hw state notifications).
> > 
> > > Also I am see very high rate for irq287/irq306 -- is this good?
> > You did not demostrated what you mean by 'very high rate'.
> > 20 req/sec does not count as 'high', and this is the highest I see
> > in your message.
> 
> Currently this is idle server and totaly interrupts count for firmware
> command completion is 50x more compared to completion queue
> interrupts. I am afraid to flooded at prodation load.
Did you reflashed firmware to the latest supported release ?
If not, do that.

Why are you sure that this is command completion and not events ?
Set sysctl dev.mce.0.conf.debug_stats=1 then look at the output
of sysctl dev.mce.0.hw_ctx_debug.

> 
> Also, completion queue interrupts may be pined to dedicaded CPU, what
> about command completion?
Command completion is normal MSI-X interrupt, but I think that the official
recommendation is to not fiddle with it.  Commands are not used for normal
tx/rx path, they only provide management and statistics.

> 
> > > 
> > > # vmstat -i | grep -e ^int -e mlx
> > > interrupt                          total       rate
> > > irq286: mlx5_core0                     1          0
> > > irq287: mlx5_core0               5135992         20
> > > irq288: mlx5_core0                     1          0
> > > irq289: mlx5_core0                 76408          0
> > > irq290: mlx5_core0                 43054          0
> > > irq291: mlx5_core0                 93826          0
> > > irq292: mlx5_core0                 39457          0
> > > irq293: mlx5_core0                 36141          0
> > > irq294: mlx5_core0                 65526          0
> > > irq295: mlx5_core0                 53399          0
> > > irq296: mlx5_core0                120885          0
> > > irq297: mlx5_core0                140690          1
> > > irq298: mlx5_core0                193578          1
> > > irq299: mlx5_core0                178332          1
> > > irq300: mlx5_core0                 75334          0
> > > irq301: mlx5_core0                207118          1
> > > irq302: mlx5_core0                108803          0
> > > irq303: mlx5_core0                 24356          0
> > > irq304: mlx5_core0                 26713          0
> > > irq305: mlx5_core1                     1          0
> > > irq306: mlx5_core1               5136296         20
> > > irq307: mlx5_core1                     1          0
> > > irq308: mlx5_core1               3634544         14
> > > irq309: mlx5_core1                 22860          0
> > > irq310: mlx5_core1                564441          2
> > > irq311: mlx5_core1                 30503          0
> > > irq312: mlx5_core1                115549          0
> > > irq313: mlx5_core1                 49815          0
> > > irq314: mlx5_core1                 10272          0
> > > irq315: mlx5_core1                 85875          0
> > > irq316: mlx5_core1                134251          1
> > > irq317: mlx5_core1                 25151          0
> > > irq318: mlx5_core1                 73376          0
> > > irq319: mlx5_core1                  5879          0
> > > irq320: mlx5_core1                 39515          0
> > > irq321: mlx5_core1                  5390          0
> > > irq322: mlx5_core1                 22726          0
> > > irq323: mlx5_core1                 60408          0