igb performance/load udp issue

Tue Dec 20 22:50:09 UTC 2011

On Tue, Dec 20, 2011 at 02:54:03PM -0700, Clint wrote:
> I'm experiencing a performance/load issue when using an Intel 82580 
> quad port NIC when trying to stream video files via NFS/UDP. The media 
> streamers are only 100Mbit, but they can bring the server to its knees 
> when trying to stream a single 90Mbit/s file that meant to test the 
> performance of the streamer. As soon as the file starts to stream one of 
> the cores is completely consumed by system/kernel, and the system 
> becomes somewhat unresponsive with one streamer active, two pretty much 
> kills the server. I've tried a bunch of different loader.conf/sysctl 

I also see similar excessive system load with em(4) on latest HEAD.
Self-clocking protocols like TCP does not seem to suffer from the
issue but UDP clearly shows the issue.  A single bulk UDP transmit
test session completely halts the box while the test is in
progress.  I have a patch that addresses it but it needs more clean
ups and Jack's review.  I'm not sure whether the patch could be
accepted since it completely removes taskqueue based approach in
TX/RX path.
Also note, multi-queue option of igb(4) can re-order UDP packets in
TX path such that it can result in poor NFS over UDP performance.
It seems TCP is more resilient on packet reodering though.

Because igb(4) shares implementation idea with em(4), it's possible
igb(4) also have the similar issue.  I have to find some spare time
to try this with igb(4) but I'm somewhat overloaded to address poor
em(4) performance on 82574 with MSI-X.

> tweaking as well as running with no tweaking at all, nothing seems to 
> help. If I replace the NIC with an old Intel NIC using the em driver all 
> of the problems disappear, and the server remains practically idle the 
> entire time. The server also has no issues when using Samba/TCP with the 
> igb driver, but the streamers can only handle 60Mbit/s via Samba. 
> Anybody have any ideas or is the igb driver really that bad?
> 
> Intel i7 920 at 3.6Ghz
> FreeBSD 8.2-STABLE w/igb driver v2.2.3
> 
> top
> last pid: 12294; load averages: 1.44, 1.24, 1.18 up 0+12:40:02 09:28:14
> 59 processes: 1 running, 58 sleeping
> CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 1: 0.0% user, 0.0% nice, 4.2% system, 0.0% interrupt, 95.8% idle
> CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 4: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle
> CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 6: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
> Mem: 30M Active, 227M Inact, 5301M Wired, 350M Free
> Swap: 4096M Total, 4096M Free
> 
>  PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
>  1800 root 128 44 0 5816K 2508K rpcsvc 4 3:11 0.00% nfsd
>  1996 root 20 44 0 116M 43708K select 2 1:21 0.00% python
>  2067 root 21 45 0 78444K 15936K uwait 1 0:29 0.00% python
>  1988 root 1 44 0 73496K 10768K select 2 0:24 0.00% snmpd
>  2229 mysql 10 44 0 55576K 8260K sigwai 0 0:12 0.00% mysqld
> 
> 
> ifconfig igb2
> igb2: flags=8943 metric 0 mtu 1500
>  options=1bb
>  media: Ethernet autoselect (1000baseT )
>  status: active
> 
> 
> netstat -I igb2
> Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
> igb2 1500  00:1b:21:d3:f5:da 20511904 0 0 76590695 0 0
> 
> 
> netstat -m
> 139376/4894/144270 mbufs in use (current/cache/total)
> 136605/3161/139766/262144 mbuf clusters in use 
> (current/cache/total/max)
> 134310/1754 mbuf+clusters out of packet secondary zone in use
> (current/cache)
> 0/306/306/12800 4k (page size) jumbo clusters in use
> (current/cache/total/max)
> 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
> 308169K/8769K/316939K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/0/0 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
> 
> 
> vmstat -i | grep igb2
> irq277: igb2:que 0 1295065 20
> irq278: igb2:que 1 64720 1
> irq279: igb2:que 2 12942365 200
> irq280: igb2:que 3 65835 1
> irq281: igb2:que 4 66044 1
> irq282: igb2:que 5 148109 2
> irq283: igb2:que 6 66648 1
> irq284: igb2:que 7 68263 1
> irq285: igb2:link 2 0