igb performance/load udp issue

Clint clint at servedomains.com
Tue Dec 20 22:09:40 UTC 2011


I'm experiencing a performance/load issue when using an Intel 82580 
quad port NIC when trying to stream video files via NFS/UDP. The media 
streamers are only 100Mbit, but they can bring the server to its knees 
when trying to stream a single 90Mbit/s file that meant to test the 
performance of the streamer. As soon as the file starts to stream one of 
the cores is completely consumed by system/kernel, and the system 
becomes somewhat unresponsive with one streamer active, two pretty much 
kills the server. I've tried a bunch of different loader.conf/sysctl 
tweaking as well as running with no tweaking at all, nothing seems to 
help. If I replace the NIC with an old Intel NIC using the em driver all 
of the problems disappear, and the server remains practically idle the 
entire time. The server also has no issues when using Samba/TCP with the 
igb driver, but the streamers can only handle 60Mbit/s via Samba. 
Anybody have any ideas or is the igb driver really that bad?

Intel i7 920 at 3.6Ghz
FreeBSD 8.2-STABLE w/igb driver v2.2.3

top
last pid: 12294; load averages: 1.44, 1.24, 1.18 up 0+12:40:02 09:28:14
59 processes: 1 running, 58 sleeping
CPU 0: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 1: 0.0% user, 0.0% nice, 4.2% system, 0.0% interrupt, 95.8% idle
CPU 2: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 3: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 4: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle
CPU 5: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 6: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
CPU 7: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 30M Active, 227M Inact, 5301M Wired, 350M Free
Swap: 4096M Total, 4096M Free

  PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
  1800 root 128 44 0 5816K 2508K rpcsvc 4 3:11 0.00% nfsd
  1996 root 20 44 0 116M 43708K select 2 1:21 0.00% python
  2067 root 21 45 0 78444K 15936K uwait 1 0:29 0.00% python
  1988 root 1 44 0 73496K 10768K select 2 0:24 0.00% snmpd
  2229 mysql 10 44 0 55576K 8260K sigwai 0 0:12 0.00% mysqld


ifconfig igb2
igb2: flags=8943 metric 0 mtu 1500
  options=1bb
  media: Ethernet autoselect (1000baseT )
  status: active


netstat -I igb2
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll
igb2 1500  00:1b:21:d3:f5:da 20511904 0 0 76590695 0 0


netstat -m
139376/4894/144270 mbufs in use (current/cache/total)
136605/3161/139766/262144 mbuf clusters in use 
(current/cache/total/max)
134310/1754 mbuf+clusters out of packet secondary zone in use
(current/cache)
0/306/306/12800 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
308169K/8769K/316939K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines


vmstat -i | grep igb2
irq277: igb2:que 0 1295065 20
irq278: igb2:que 1 64720 1
irq279: igb2:que 2 12942365 200
irq280: igb2:que 3 65835 1
irq281: igb2:que 4 66044 1
irq282: igb2:que 5 148109 2
irq283: igb2:que 6 66648 1
irq284: igb2:que 7 68263 1
irq285: igb2:link 2 0


More information about the freebsd-net mailing list