High CPU interrupt load on intel I350T4 with igb on 8.3

"Clément Hermann (nodens)" nodens2099 at gmail.com
Fri Apr 26 11:30:45 UTC 2013 

Hi list,

We use pf+ALTQ for trafic shaping on some routers.

We are switching to new servers : Dell PowerEdge R620 with 2 8-cores 
Intel Processor (E5-2650L), 8GB RAM and Intel I350T4 (quad port) using 
igb driver. The old hardware is using em driver, the CPU load is high 
but mostly due to kernel and a large pf ruleset.

On the new hardware, we see high CPU Interrupt load (up to 95%), even 
though there is not much trafic currently (peaks about 150Mbps and 
40Kpps). All queues are used and binded to a cpu according to top, but a 
lot of CPU time is spent on igb queues (interrupt or wait). The load is 
fine when we stay below 20Kpps.

We see no mbuf shortage, no dropped packet, but there is little margin 
left on CPU time (about 25% idle at best, most of CPU time is spent on 
interrupts), which is disturbing.

We have done some tuning, but to no avail :

sysctl.conf :

# mbufs
kern.ipc.nmbclusters=65536
# Sockets
kern.ipc.somaxconn=8192
net.inet.tcp.delayed_ack=0
net.inet.tcp.sendspace=65535
net.inet.udp.recvspace=65535
net.inet.udp.maxdgram=57344
net.local.stream.recvspace=65535
net.local.stream.sendspace=65535
# IGB
dev.igb.0.rx_processing_limit=4096
dev.igb.1.rx_processing_limit=4096
dev.igb.2.rx_processing_limit=4096
dev.igb.3.rx_processing_limit=4096

/boot/loader.conf :

vm.kmem_size=1G
hw.igb.max_interrupt_rate="32000"  # maximum number of interrupts/sec 
generated by single igb(4) (default 8000)
hw.igb.txd="2048"                              # number of transmit 
descriptors allocated by the driver (2048 limit)
hw.igb.rxd="2048"                              # number of receive 
descriptors allocated by the driver (2048 limit)
hw.igb.rx_process_limit="1000"     # maximum number of received packets 
to process at a time, The default of 100 is
                                                                # too 
low for most firewalls. (-1 means unlimited)

Kernel HZ is 1000.

The IGB /boot/loader.conf tuning was our last attempt, it didn't change 
anything.

Does anyone have any pointer ? How could we lower CPU interrupt load ? 
should we set hw.igb.max_interrupt_rate lower instead of higher ?
 From what we saw here and there, we should be able to do much better 
with this hardware.


relevant sysctl (igb1 and igb2 only, other interfaces are unused) :

sysctl dev.igb | grep -v ": 0$"
dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 2.3.1
dev.igb.1.%driver: igb
dev.igb.1.%location: slot=0 function=1
dev.igb.1.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x8086 
subdevice=0x5001 class=0x020000
dev.igb.1.%parent: pci5
dev.igb.1.nvm: -1
dev.igb.1.enable_aim: 1
dev.igb.1.fc: 3
dev.igb.1.rx_processing_limit: 4096
dev.igb.1.eee_disabled: 1
dev.igb.1.link_irq: 2
dev.igb.1.device_control: 1209795137
dev.igb.1.rx_control: 67141658
dev.igb.1.interrupt_mask: 4
dev.igb.1.extended_int_mask: 2147483981
dev.igb.1.fc_high_water: 33168
dev.igb.1.fc_low_water: 33152
dev.igb.1.queue0.interrupt_rate: 71428
dev.igb.1.queue0.txd_head: 1318
dev.igb.1.queue0.txd_tail: 1318
dev.igb.1.queue0.tx_packets: 84663594
dev.igb.1.queue0.rxd_head: 717
dev.igb.1.queue0.rxd_tail: 715
dev.igb.1.queue0.rx_packets: 43899597
dev.igb.1.queue0.rx_bytes: 8905556030
dev.igb.1.queue1.interrupt_rate: 90909
dev.igb.1.queue1.txd_head: 693
dev.igb.1.queue1.txd_tail: 693
dev.igb.1.queue1.tx_packets: 57543349
dev.igb.1.queue1.rxd_head: 1033
dev.igb.1.queue1.rxd_tail: 1032
dev.igb.1.queue1.rx_packets: 54821897
dev.igb.1.queue1.rx_bytes: 9944955108
dev.igb.1.queue2.interrupt_rate: 100000
dev.igb.1.queue2.txd_head: 350
dev.igb.1.queue2.txd_tail: 350
dev.igb.1.queue2.tx_packets: 62320990
dev.igb.1.queue2.rxd_head: 1962
dev.igb.1.queue2.rxd_tail: 1939
dev.igb.1.queue2.rx_packets: 43909016
dev.igb.1.queue2.rx_bytes: 8673941461
dev.igb.1.queue3.interrupt_rate: 14925
dev.igb.1.queue3.txd_head: 647
dev.igb.1.queue3.txd_tail: 647
dev.igb.1.queue3.tx_packets: 58776199
dev.igb.1.queue3.rxd_head: 692
dev.igb.1.queue3.rxd_tail: 691
dev.igb.1.queue3.rx_packets: 55138996
dev.igb.1.queue3.rx_bytes: 9310217354
dev.igb.1.queue4.interrupt_rate: 100000
dev.igb.1.queue4.txd_head: 1721
dev.igb.1.queue4.txd_tail: 1721
dev.igb.1.queue4.tx_packets: 54337209
dev.igb.1.queue4.rxd_head: 1609
dev.igb.1.queue4.rxd_tail: 1598
dev.igb.1.queue4.rx_packets: 46546503
dev.igb.1.queue4.rx_bytes: 8818182840
dev.igb.1.queue5.interrupt_rate: 11627
dev.igb.1.queue5.txd_head: 254
dev.igb.1.queue5.txd_tail: 254
dev.igb.1.queue5.tx_packets: 53117182
dev.igb.1.queue5.rxd_head: 701
dev.igb.1.queue5.rxd_tail: 685
dev.igb.1.queue5.rx_packets: 43014837
dev.igb.1.queue5.rx_bytes: 8699057447
dev.igb.1.queue6.interrupt_rate: 55555
dev.igb.1.queue6.txd_head: 8
dev.igb.1.queue6.txd_tail: 8
dev.igb.1.queue6.tx_packets: 52654088
dev.igb.1.queue6.rxd_head: 1057
dev.igb.1.queue6.rxd_tail: 1041
dev.igb.1.queue6.rx_packets: 45227030
dev.igb.1.queue6.rx_bytes: 9494489640
dev.igb.1.queue7.interrupt_rate: 5235
dev.igb.1.queue7.txd_head: 729
dev.igb.1.queue7.txd_tail: 729
dev.igb.1.queue7.tx_packets: 61926105
dev.igb.1.queue7.rxd_head: 146
dev.igb.1.queue7.rxd_tail: 140
dev.igb.1.queue7.rx_packets: 51781775
dev.igb.1.queue7.rx_bytes: 8901279226
dev.igb.1.mac_stats.missed_packets: 1657
dev.igb.1.mac_stats.recv_no_buff: 405
dev.igb.1.mac_stats.total_pkts_recvd: 384332760
dev.igb.1.mac_stats.good_pkts_recvd: 384331103
dev.igb.1.mac_stats.bcast_pkts_recvd: 15510
dev.igb.1.mac_stats.mcast_pkts_recvd: 52957
dev.igb.1.mac_stats.rx_frames_64: 195496498
dev.igb.1.mac_stats.rx_frames_65_127: 133346124
dev.igb.1.mac_stats.rx_frames_128_255: 5254911
dev.igb.1.mac_stats.rx_frames_256_511: 9700049
dev.igb.1.mac_stats.rx_frames_512_1023: 16885886
dev.igb.1.mac_stats.rx_frames_1024_1522: 23647635
dev.igb.1.mac_stats.good_octets_recvd: 74284029276
dev.igb.1.mac_stats.good_octets_txd: 544536708502
dev.igb.1.mac_stats.total_pkts_txd: 485327419
dev.igb.1.mac_stats.good_pkts_txd: 485327419
dev.igb.1.mac_stats.bcast_pkts_txd: 72
dev.igb.1.mac_stats.mcast_pkts_txd: 52820
dev.igb.1.mac_stats.tx_frames_64: 57820809
dev.igb.1.mac_stats.tx_frames_65_127: 51586341
dev.igb.1.mac_stats.tx_frames_128_255: 7050579
dev.igb.1.mac_stats.tx_frames_256_511: 7887126
dev.igb.1.mac_stats.tx_frames_512_1023: 10130891
dev.igb.1.mac_stats.tx_frames_1024_1522: 350851673
dev.igb.1.interrupts.asserts: 551135045
dev.igb.1.interrupts.rx_pkt_timer: 384326679
dev.igb.1.interrupts.tx_queue_empty: 485323376
dev.igb.1.interrupts.tx_queue_min_thresh: 6324386
dev.igb.1.host.rx_pkt: 4424
dev.igb.1.host.tx_good_pkt: 4043
dev.igb.1.host.rx_good_bytes: 74284030864
dev.igb.1.host.tx_good_bytes: 544536708502
dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection version - 2.3.1
dev.igb.2.%driver: igb
dev.igb.2.%location: slot=0 function=2
dev.igb.2.%pnpinfo: vendor=0x8086 device=0x1521 subvendor=0x8086 
subdevice=0x5001 class=0x020000
dev.igb.2.%parent: pci5
dev.igb.2.nvm: -1
dev.igb.2.enable_aim: 1
dev.igb.2.fc: 3
dev.igb.2.rx_processing_limit: 4096
dev.igb.2.eee_disabled: 1
dev.igb.2.link_irq: 2
dev.igb.2.device_control: 1209795137
dev.igb.2.rx_control: 67141658
dev.igb.2.interrupt_mask: 4
dev.igb.2.extended_int_mask: 2147483959
dev.igb.2.fc_high_water: 33168
dev.igb.2.fc_low_water: 33152
dev.igb.2.queue0.interrupt_rate: 13698
dev.igb.2.queue0.txd_head: 1618
dev.igb.2.queue0.txd_tail: 1618
dev.igb.2.queue0.tx_packets: 46401106
dev.igb.2.queue0.rxd_head: 831
dev.igb.2.queue0.rxd_tail: 827
dev.igb.2.queue0.rx_packets: 69356350
dev.igb.2.queue0.rx_bytes: 68488772907
dev.igb.2.queue1.interrupt_rate: 5405
dev.igb.2.queue1.txd_head: 190
dev.igb.2.queue1.txd_tail: 190
dev.igb.2.queue1.tx_packets: 55965886
dev.igb.2.queue1.rxd_head: 268
dev.igb.2.queue1.rxd_tail: 256
dev.igb.2.queue1.rx_packets: 58958084
dev.igb.2.queue1.rx_bytes: 69154569937
dev.igb.2.queue2.interrupt_rate: 83333
dev.igb.2.queue2.txd_head: 568
dev.igb.2.queue2.txd_tail: 568
dev.igb.2.queue2.tx_packets: 44974648
dev.igb.2.queue2.rxd_head: 371
dev.igb.2.queue2.rxd_tail: 219
dev.igb.2.queue2.rx_packets: 67037407
dev.igb.2.queue2.rx_bytes: 72042326102
dev.igb.2.queue3.interrupt_rate: 12658
dev.igb.2.queue3.txd_head: 867
dev.igb.2.queue3.txd_tail: 867
dev.igb.2.queue3.tx_packets: 55962467
dev.igb.2.queue3.rxd_head: 85
dev.igb.2.queue3.rxd_tail: 1953
dev.igb.2.queue3.rx_packets: 60972965
dev.igb.2.queue3.rx_bytes: 70397176035
dev.igb.2.queue4.interrupt_rate: 90909
dev.igb.2.queue4.txd_head: 1920
dev.igb.2.queue4.txd_tail: 1920
dev.igb.2.queue4.tx_packets: 47660931
dev.igb.2.queue4.rxd_head: 1397
dev.igb.2.queue4.rxd_tail: 1379
dev.igb.2.queue4.rx_packets: 59110758
dev.igb.2.queue4.rx_bytes: 68919201478
dev.igb.2.queue5.interrupt_rate: 111111
dev.igb.2.queue5.txd_head: 886
dev.igb.2.queue5.txd_tail: 886
dev.igb.2.queue5.tx_packets: 45103990
dev.igb.2.queue5.rxd_head: 812
dev.igb.2.queue5.rxd_tail: 799
dev.igb.2.queue5.rx_packets: 59030312
dev.igb.2.queue5.rx_bytes: 69234293962
dev.igb.2.queue6.interrupt_rate: 5208
dev.igb.2.queue6.txd_head: 1926
dev.igb.2.queue6.txd_tail: 1926
dev.igb.2.queue6.tx_packets: 46215046
dev.igb.2.queue6.rxd_head: 692
dev.igb.2.queue6.rxd_tail: 689
dev.igb.2.queue6.rx_packets: 58256050
dev.igb.2.queue6.rx_bytes: 68429172749
dev.igb.2.queue7.interrupt_rate: 50000
dev.igb.2.queue7.txd_head: 126
dev.igb.2.queue7.txd_tail: 126
dev.igb.2.queue7.tx_packets: 52451455
dev.igb.2.queue7.rxd_head: 968
dev.igb.2.queue7.rxd_tail: 885
dev.igb.2.queue7.rx_packets: 65946491
dev.igb.2.queue7.rx_bytes: 70263478849
dev.igb.2.mac_stats.missed_packets: 958
dev.igb.2.mac_stats.recv_no_buff: 69
dev.igb.2.mac_stats.total_pkts_recvd: 498658079
dev.igb.2.mac_stats.good_pkts_recvd: 498657121
dev.igb.2.mac_stats.bcast_pkts_recvd: 16867
dev.igb.2.mac_stats.mcast_pkts_recvd: 52957
dev.igb.2.mac_stats.rx_frames_64: 59089332
dev.igb.2.mac_stats.rx_frames_65_127: 52880118
dev.igb.2.mac_stats.rx_frames_128_255: 7526966
dev.igb.2.mac_stats.rx_frames_256_511: 8468389
dev.igb.2.mac_stats.rx_frames_512_1023: 10434770
dev.igb.2.mac_stats.rx_frames_1024_1522: 360257545
dev.igb.2.mac_stats.good_octets_recvd: 558910494322
dev.igb.2.mac_stats.good_octets_txd: 84618858153
dev.igb.2.mac_stats.total_pkts_txd: 394726904
dev.igb.2.mac_stats.good_pkts_txd: 394726904
dev.igb.2.mac_stats.bcast_pkts_txd: 48
dev.igb.2.mac_stats.mcast_pkts_txd: 52821
dev.igb.2.mac_stats.tx_frames_64: 196605932
dev.igb.2.mac_stats.tx_frames_65_127: 134602807
dev.igb.2.mac_stats.tx_frames_128_255: 5705236
dev.igb.2.mac_stats.tx_frames_256_511: 10267168
dev.igb.2.mac_stats.tx_frames_512_1023: 17165496
dev.igb.2.mac_stats.tx_frames_1024_1522: 30380265
dev.igb.2.interrupts.asserts: 465994260
dev.igb.2.interrupts.rx_pkt_timer: 498647546
dev.igb.2.interrupts.tx_queue_empty: 394720657
dev.igb.2.interrupts.tx_queue_min_thresh: 24424555
dev.igb.2.host.rx_pkt: 9575
dev.igb.2.host.tx_good_pkt: 6248
dev.igb.2.host.rx_good_bytes: 558910513984
dev.igb.2.host.tx_good_bytes: 84618858217


Thanks for your help.

Cheers,

-- 
Clement (nodens)

More information about the freebsd-net mailing list