High interrupt rate on a PF box + performance

[Damien Fleuriot]

On 1/27/11 6:37 PM, Jeremy Chadwick wrote:
> On Thu, Jan 27, 2011 at 06:31:29PM +0100, Damien Fleuriot wrote:
>>
>>
>> On 1/27/11 6:27 PM, Jeremy Chadwick wrote:
>>> On Thu, Jan 27, 2011 at 10:57:14AM +0100, Damien Fleuriot wrote:
>>>> Hello list,
>>>>
>>>> I have a problem with interrupts, network cards, and PF performance.
>>>>
>>>> We have 2 firewalls running FreeBSD 8.0 for the current master and
>>>> FreeBSD 8.1 for the backup host, which I upgraded just yesterday.
>>>>
>>>> [...]
>>>>
>>>> vmstat -i
>>>> ---
>>>> interrupt                          total       rate
>>>> irq14: ata0                           36          0
>>>> irq16: mfi0                       353244          1
>>>> irq21: uhci0 uhci+                461504          1
>>>> cpu0: timer                    615183815       1996
>>>> irq256: bce0                  1015412475       3295
>>>> irq257: igb0                  1067318584       3464
>>>> irq258: igb0                   695648752       2258
>>>> irq259: igb0                           2          0
>>>> irq260: igb1                    11503857         37
>>>> irq261: igb1                      506598          1
>>>> irq262: igb1                          69          0
>>>> irq269: bce1                      790820          2
>>>> cpu1: timer                    615183757       1996
>>>> cpu2: timer                    615197165       1996
>>>> cpu3: timer                    615197165       1996
>>>> Total                         5252757843      17050
>>>
>>> There are changes to the igb(4) driver which are in RELENG_8 (8-STABLE),
>>> and some which will be in the upcoming 8.2-RELEASE, which may address
>>> this.  Jack Vogel of Intel would be able to confirm for sure; CC'ing him
>>> here.
>>>
>>> Could you please provide output from the following commands?
>>>
>>> * pciconf -lvcb  (only include igbX entries, thanks)
>>> * sysctl -a | grep msi
>>>
>>> Thanks.
>>>
>>> I can't help with the CARP-related issues or other stuff you're
>>> experiencing.  These issues may all be separate problems, hard to say.
>>>
>>
>
> What you did here was "pciconf -lvcb | grep igb", or something
> equivalent.

Indeed, my bad.

igb0 at pci0:14:0:0:	class=0x020000 card=0x145a8086 chip=0x10d68086
rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82575GB Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xfd0e0000, size 131072,
enabled
    bar   [14] = type Memory, range 32, base 0xfce00000, size 2097152,
enabled
    bar   [18] = type I/O Port, range 32, base 0xdce0, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xfd0dc000, size 16384, enabled
    cap 01[40] = powerspec 2  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 11[60] = MSI-X supports 10 messages in map 0x1c enabled
    cap 10[a0] = PCI-Express 2 endpoint max data 256(256) link x4(x4)
igb1 at pci0:14:0:1:	class=0x020000 card=0x145a8086 chip=0x10d68086
rev=0x02 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82575GB Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xfd0a0000, size 131072,
enabled
    bar   [14] = type Memory, range 32, base 0xfcc00000, size 2097152,
enabled
    bar   [18] = type I/O Port, range 32, base 0xdcc0, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xfd0d8000, size 16384, enabled
    cap 01[40] = powerspec 2  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 11[60] = MSI-X supports 10 messages in map 0x1c enabled
    cap 10[a0] = PCI-Express 2 endpoint max data 256(256) link x4(x4)