NFS over LAGG / lacp poor performance

[Steven Hartland]

----- Original Message ----- 
From: "Gerrit Kühn" <gerrit.kuehn at aei.mpg.de>
To: "Steven Hartland" <killing at multiplay.co.uk>
Cc: <freebsd-net at freebsd.org>
Sent: Friday, April 25, 2014 3:13 PM
Subject: Re: NFS over LAGG / lacp poor performance


> On Fri, 25 Apr 2014 15:02:15 +0100 "Steven Hartland"
> <killing at multiplay.co.uk> wrote about Re: NFS over LAGG / lacp poor
> performance:
>
> SH> We find that large numbers of queues causes high interrupt issues
>
> Like the thing I am seeing with igb1 on my system?
>
> ---
> root at storage:/root # vmstat -i
> interrupt                          total       rate
> irq1: atkbd0                        2743          0
> irq18: ehci0 uhci5               4445560          2
> irq21: uhci1                          29          0
> cpu0:timer                     355724275        227
> irq256: igb0:que 0              99437514         63
> irq257: igb0:que 1              61534816         39
> irq258: igb0:que 2             101725601         65
> irq259: igb0:que 3             100864440         64
> irq260: igb0:link                      2          0
> irq261: igb1:que 0               1689527          1
> irq262: igb1:que 1            2357590958       1510
> irq263: igb1:que 2               1584474          1
> irq264: igb1:que 3               1923144          1
> irq265: igb1:link                      2          0
> irq266: mps0                   332232450        212
> irq267: mps1                   194207894        124
> irq268: mps2                   176700834        113
> irq269: ahci0                   59175548         37
> cpu1:timer                     419838321        268
> cpu3:timer                     329696415        211
> cpu2:timer                     328219053        210
> Total                         4926593600       3156
> ---
>
>
> irq262 sticks out like a sore thumb...
>
> SH> however at a guess you did this to enable the machine to boot with
> SH> all nics due to lack of auto mbuf tuning in 9.x.
> SH> I'd go with ~2 queues per nic.
>
> I was wondering what to try next for my system: either manually set the
> queues back to 2 or 1 per NIC, or try upgrading to either 9.2 or 10 as it
> looked like there have been improvements in the igb driver. Do you have
> any recommendations on that?

We saw the issue with 10-RELEASE last weekend on a machine with 6 x igb's
where by the system was burning CPU in the interrupt handlers, setting
num_queues to 2 fixed the issue we where seeing.

    Regards
    Steve