NFS over LAGG / lacp poor performance

[Steven Hartland]

----- Original Message ----- 
From: "Marek Salwerowicz" <marek_sal at wp.pl>
To: "Steven Hartland" <killing at multiplay.co.uk>; "Gerrit Kühn" <gerrit.kuehn at aei.mpg.de>
Cc: <freebsd-net at freebsd.org>
Sent: Friday, April 25, 2014 2:06 PM
Subject: Re: NFS over LAGG / lacp poor performance


>W dniu 2014-04-25 14:55, Steven Hartland pisze:
>> ----- Original Message ----- From: "Marek Salwerowicz" <marek_sal at wp.pl>
>>
>>
>>> W dniu 2014-04-25 14:01, Gerrit Kühn pisze:
>>>> Thanks for your input. As far as I understood so far, there should
>>>> be one
>>>> igb queue created per cpu core in the system by default (and this is
>>>> what
>>>> I see on my system). But my irq rate looks quite high to me (and it is
>>>> only on one of these queues).
>>>
>>>
>>> My CPU has 8 cores:
>>>
>>> http://ark.intel.com/products/75267/Intel-Xeon-Processor-E5-2640-v2-20M-Cache-2_00-GHz
>>>
>>>
>>> So why do I have only 1 queue ?
>>
>> What does "sysctl hw.igb.num_queues" report?
>
> storage1% sysctl hw.igb.num_queues
> hw.igb.num_queues: 1
>>
>> num_queues does default to 1 for Legacy or MSI so you might be hitting
>> that.
>>
>> Do you see "Using MSIX interrupts with" in your dmesg?
> storage% dmesg | grep MSIX
> igb0: Using MSIX interrupts with 2 vectors
> igb1: Using MSIX interrupts with 2 vectors
> igb2: Using MSIX interrupts with 2 vectors
> igb3: Using MSIX interrupts with 2 vectors
> igb0: Using MSIX interrupts with 2 vectors
> igb1: Using MSIX interrupts with 2 vectors
> igb2: Using MSIX interrupts with 2 vectors
> igb3: Using MSIX interrupts with 2 vectors

In that case I believe you've hard coded the number of queues, check /boot/loader.conf
for references to this.

    Regards
    Steve