High interrupt rate on a PF box + performance

[Damien Fleuriot]

On 1/28/11 11:29 AM, Damien Fleuriot wrote:
> On 1/27/11 10:44 PM, Jack Vogel wrote:
>>
>> The 8.X kernel is NOT single-threaded. Anything but. And the stack has
>> also been improved, I believe there are still bottlenecks but its far better
>> than the old days.
>>
>> The igb driver in 8.2 creates up to 8 queues on the right hardware, they
>> are each auto-bound to a particular CPU.
>>
>> The older version you are running had issues and hence multiqueue was
>> not enabled.  So, do upgrade once 8.2 is finalized :)
>>
>> Cheers,
>>
>> Jack
>>
> 
> Going to push for us to install 8.2 as soon as the release hits, thanks
> for your feedback Jack :)

Hello guys, list,



This is a quick headsup regarding this issue.

We have now swapped our PF firewalls to active-active and observed, as
one would expect, approx. 50% drop of traffic, seeing it's now balanced
between 2 machines :)

We have also disabled pfsync (which also resulted in a massive drop of
interrupts).



One of the hosts is running 8.2-PRERELEASE , and this is the one for
which I'm providing stats now.

For completeness, also find the graphs:
http://my.gd/fw_graphs/



# vmstat -i
interrupt                          total       rate
irq16: mpt0                       320899          0
irq21: atapci1                        35          0
irq22: ehci0 ehci1               1992267          2
cpu0: timer                   1330310985       1979
irq258: igb0:que 0                829898          1
irq259: igb0:que 1                  3255          0
irq260: igb0:que 2                  2059          0
irq261: igb0:que 3                  1060          0
irq262: igb0:link                      2          0
irq263: igb1:que 0            2676083520       3981
irq264: igb1:que 1            2676853656       3982
irq265: igb1:que 2            2682493388       3990
irq266: igb1:que 3            2688637571       3999
irq267: igb1:link                      2          0
irq273: igb3:que 0            2654678899       3949
irq274: igb3:que 1            2648682488       3940
irq275: igb3:que 2            2650599952       3943
irq276: igb3:que 3            2657367887       3953
irq277: igb3:link                      2          0
cpu1: timer                   1330301807       1978
cpu2: timer                   1330301315       1978
cpu3: timer                   1330301347       1978
Total                        26659762294      39659





# pfctl -si
Status: Enabled for 7 days 18:43:34           Debug: Urgent

Interface Stats for igb3              IPv4             IPv6
  Bytes In                   1585211309166                0
  Bytes Out                  2044715081803                0
  Packets In
    Passed                      6238056055                0
    Blocked                       15350206                0
  Packets Out
    Passed                      6300823415                0
    Blocked                        1223577                0

State Table                          Total             Rate
  current entries                    37627
  searches                     25108284353        37351.6/s
  inserts                       2157108574         3209.0/s
  removals                      2157070947         3208.9/s
Counters
  match                         2175657232         3236.6/s
  bad-offset                             0            0.0/s
  fragment                             104            0.0/s
  short                                  5            0.0/s
  normalize                            557            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              6            0.0/s
  proto-cksum                        52649            0.1/s
  state-mismatch                    340029            0.5/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                             90            0.0/s





igb0 at pci0:7:0:0:	class=0x020000 card=0x145a8086 chip=0x10d68086 rev=0x02
hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82575GB Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xdabc0000, size 131072,
enabled
    bar   [14] = type Memory, range 32, base 0xdac00000, size 2097152,
enabled
    bar   [18] = type I/O Port, range 32, base 0xdcc0, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xdabb8000, size 16384, enabled
    cap 01[40] = powerspec 2  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit
    cap 11[60] = MSI-X supports 10 messages in map 0x1c enabled
    cap 10[a0] = PCI-Express 2 endpoint max data 256(256) link x4(x4)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 001b21ffff12f438
  synproxy                               0            0.0/s


(there are 4 of these, it's a quad port card)