Intel x520-t2: interrupt storm detected + throttling

[Iordan Iordanov]

Hello,

I decided to send a separate message about another problem that we 
witnessed while testing 10GbaseT connectivity between a pair of Intel 
x520-t2 interconnected with cat6 cable.

While we were testing we got a multitude of kernel messages of this form:

interrupt storm detected on "irq271:"; throttling interrupt source
interrupt storm detected on "irq271:"; throttling interrupt source
interrupt storm detected on "irq271:"; throttling interrupt source

An easy way for us to reproduce the problem was this. Start an nc 
process listening on one machine:

fw4# nc -l 8080 > /dev/null

Then, start piping from /dev/zero into nc from the other machine:

fw3# dd if=/dev/zero bs=4096k | nc 192.168.1.200 8080

Almost immediately, on the server-side, we start getting the messages. 
The client-side is silent. When I reversed the client/server roles, the 
scenario was repeated the other way around to prove to myself it's the 
server that reports the issue. The transfer speed for a single process 
over TCP appears to be about 240 megabytes/s or about 2 gigabits/s. 
However, with multiple nc processes we are able to push this to about 4 
gigabits/s (similar to our iperf tests).

A curious, and probably easy to explain fact is that when doing 
transfers over UDP (the -u flag to nc), we do not get the "interrupt 
storm" messages, and indeed, the interrupt rate in "top" is 1/2 that of 
when we do the TCP test under the same conditions.

More information about our hardware is:

Motherboard: Supermicro H8DGU-F
CPU:         AMD 6128 8-core
Memory:      16GB DDR3
NICs:        Intel x520-t2 10GbaseT adapters

An ifconfig ix1 output for one of the machines:

fw4# ifconfig ix1
ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500

options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO>
         ether 00:1b:21:d6:19:51
         inet 192.168.1.200 netmask 0xffffff00 broadcast 192.168.1.255
         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
         media: Ethernet autoselect (autoselect <full-duplex>)
         status: active


Many thanks for any help/action on this!

Sincerely,

Iordan Iordanov
CDF System Administrator
University of Toronto