10Gb/s ixgbe and mbuf performance tuning.

Mon May 14 16:50:29 UTC 2012

Hi,

                I am running FreeBSD 8.1p3 amd64 with 6Gb of RAM. This
is a router using Quagga for BGP.  It has two 10Gb/s network connections
using the intel ixgbe 2.4.4 driver.  It is handling routing for about
200 remote sites and about 300k simultaneous sessions through the box.
After a few hours at high traffic volumes (just over a 1Gb/s) it will
stop routing any traffic.  It look like I am running out of mbuf's.  I
have increased them but I am worried that there is a limit as to how
much I can increase them without starving the kernel of space.  Console
is still responsive but no network.

                I have looked over Tuning(7) and some other great
articles on the net on performance tuning FreeBSD but it has left me a
little confused.  Can someone help to explain the best way to
troubleshoot this issue and any formulas to tune it.

                Here is some information:

# netstat -m

66042/18063/84105 mbufs in use (current/cache/total)

66039/15931/81970/524288 mbuf clusters in use (current/cache/total/max)

66039/15881 mbuf+clusters out of packet secondary zone in use
(current/cache)

0/5/5/524288 4k (page size) jumbo clusters in use
(current/cache/total/max)

0/0/0/131072 9k jumbo clusters in use (current/cache/total/max)

0/0/0/65536 16k jumbo clusters in use (current/cache/total/max)

148588K/36397K/184986K bytes allocated to network (current/cache/total)

0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)

0/0/0 requests for jumbo clusters denied (4k/9k/16k)

0/0/0 sfbufs in use (current/peak/max)

0 requests for sfbufs denied

0 requests for sfbufs delayed

0 requests for I/O initiated by sendfile

0 calls to protocol drain routines

# more /boot/loader.conf

geom_mirror_load="YES"

loader_logo="beastie"

vesa_load="YES"

ixgbe_load="YES"

hw.ixgbe.txd=4096

hw.ixgbe.rxd=4096

kern.ipc.nmbjumbop=524288

kern.ipc.nmbclusters=524288

kern.ipc.maxsockbuf=4194304

# more /etc/sysctl.conf 

#security.bsd.see_other_uids=0

kern.ipc.nmbjumbop=524288

kern.ipc.nmbclusters=524288

kern.ipc.maxsockbuf=4194304

hw.intr_storm_threshold=9000

net.inet.icmp.icmplim=1000

Copyright (c) 1992-2010 The FreeBSD Project.

Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

        The Regents of the University of California. All rights
reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.

FreeBSD 8.1-RELEASE-p1 #3: Thu Mar 29 08:53:27 MDT 2012

    root at SchoolGW2.epsb.ca:/usr/obj/usr/src/sys/DELLR710x64 amd64

Timecounter "i8254" frequency 1193182 Hz quality 0

CPU: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz (2793.02-MHz
K8-class CPU)

  Origin = "GenuineIntel"  Id = 0x106a5  Family = 6  Model = 1a
Stepping = 5

Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

Features2=0x9ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PD
CM,DCA,SSE4.1,SSE4.2,POPCNT>

  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>

  AMD Features2=0x1<LAHF>

  TSC: P-state invariant

real memory  = 6442450944 (6144 MB)

avail memory = 6170185728 (5884 MB)

ACPI APIC Table: <DELL   PE_SC3  >

FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs

FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads

cpu0 (BSP): APIC ID: 16

cpu1 (AP): APIC ID: 17

cpu2 (AP): APIC ID: 18

cpu3 (AP): APIC ID: 19

cpu4 (AP): APIC ID: 20

cpu5 (AP): APIC ID: 21

cpu6 (AP): APIC ID: 22

cpu7 (AP): APIC ID: 23

cpu8 (AP): APIC ID:  0

cpu9 (AP): APIC ID:  1

cpu10 (AP): APIC ID:  2

cpu11 (AP): APIC ID:  3

cpu12 (AP): APIC ID:  4

cpu13 (AP): APIC ID:  5

cpu14 (AP): APIC ID:  6

cpu15 (AP): APIC ID:  7

....

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4>
port 0xbcc0-0xbcdf mem 0xdd800000-0xdd9fffff,0xdd7f8000-0xdd7fbfff irq
50 at device 0.0 on pci8

ix0: Using MSIX interrupts with 9 vectors

ix0: [ITHREAD] 

ix0: [ITHREAD] 

ix0: [ITHREAD] 

ix0: [ITHREAD]

ix0: [ITHREAD]

ix0: [ITHREAD]

ix0: [ITHREAD] 

ix0: [ITHREAD] 

ix0: [ITHREAD] 

ix0: Ethernet address: 00:1b:21:d8:d8:be

ix0: PCI Express Bus: Speed 5.0Gb/s Width x8

ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.4.4>
port 0xbce0-0xbcff mem 0xdda00000-0xddbfffff,0xdd7fc000-0xdd7fffff irq
40 at device 0.1 on pci8

ix1: Using MSIX interrupts with 9 vectors

ix1: [ITHREAD] 

ix1: [ITHREAD] 

ix1: [ITHREAD] 

ix1: [ITHREAD]

ix1: [ITHREAD]

ix1: [ITHREAD]

ix1: [ITHREAD]

ix1: [ITHREAD]

ix1: [ITHREAD] 

ix1: Ethernet address: 00:1b:21:d8:d8:bf

ix1: PCI Express Bus: Speed 5.0Gb/s Width x8

# ifconfig -a

ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500

options=405b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLA
N_HWTSO>

        ether 00:1b:21:d8:d8:be

        inet 192.168.248.228 netmask 0xffffffe0 broadcast
192.168.248.255

        media: Ethernet autoselect (10Gbase-T <full-duplex>)

        status: active

ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500

options=405b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,LRO,VLA
N_HWTSO>

        ether 00:1b:21:d8:d8:bf

        inet 192.168.250.40 netmask 0xffffff00 broadcast 192.168.250.255

        inet 192.168.250.23 netmask 0xffffff00 broadcast 192.168.250.255

        media: Ethernet autoselect (10Gbase-T <full-duplex>)

        status: active

---- Kirk

Kirk Davis

Supervisor - Network Operations

ITS, Edmonton Public Schools

One Kingsway Ave. NW

Edmonton, Alberta

T5H 4G9

(780) 429-8308