kern/130059: Leaking 50k mbufs/hour

Jason DiCioccio jd at
Tue Dec 30 18:00:10 UTC 2008

>Number:         130059
>Category:       kern
>Synopsis:       Leaking 50k mbufs/hour
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Tue Dec 30 18:00:09 UTC 2008
>Originator:     Jason DiCioccio
>Release:        FreeBSD 7.1-RC2 i386
System: FreeBSD 7.1-RC2 FreeBSD 7.1-RC2 #5: Sat Dec 27 21:16:55 EST 2008 geniusj at i386

FreeBSD 7.1-RC2 #5: Sat Dec 27 21:16:55 EST 2008
    geniusj at
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) 4 CPU 2.80GHz (2800.11-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0xf29  Stepping = 9
real memory  = 1071906816 (1022 MB)
avail memory = 1039556608 (991 MB)
acpi0: <INTEL D845GVS1> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 3fe00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf7ffffff,0xffa80000-0xffafffff irq 11 at device 2.0 on pci0
agp0: <Intel 82845M (845M GMCH) SVGA controller> on vgapci0
agp0: detected 892k stolen memory
agp0: aperture size is 128M
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pci0: <serial bus, USB> at device 29.7 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib1
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xdc00-0xdc3f mem 0xff8e0000-0xff8fffff,0xff8c0000-0xff8dffff irq 7 at device 1.0 on pci1
em0: [FILTER]
em0: Ethernet address: 00:1b:21:26:13:f2
pci1: <network, ethernet> at device 8.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH4 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Sleep Button> on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
cryptosoft0: <software crypto> on motherboard
orm0: <ISA Option ROMs> at iomem 0xcb800-0xcc7ff,0xcc800-0xcd7ff,0xcd800-0xce7ff pnpid ORM0000 on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 8250 or not responding
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2800112132 Hz quality 800
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
ad0: 76319MB <WDC WD800BB-00FJA0 13.03G13> at ata0-master UDMA100
Trying to mount root from ufs:/dev/ad0s1a

machine		i386
cpu		I686_CPU
ident		UPDATE

# To statically compile in device wiring instead of /boot/device.hints
#hints		"GENERIC.hints"		# Default places to look for devices.

makeoptions	DEBUG=-g		# Build kernel with gdb(1) debug symbols

options 	SCHED_ULE		# 4BSD scheduler
options 	PREEMPTION		# Enable kernel thread preemption
options 	INET			# InterNETworking
options 	INET6			# IPv6 communications protocols
options		IPSEC
options 	FFS			# Berkeley Fast Filesystem
options		GEOM_PART_GPT		# GUID Partition Tables.
options		GEOM_LABEL		# GEOM Labelization
options 	SOFTUPDATES		# Enable FFS soft updates support
options 	UFS_ACL			# Support for access control lists
options 	UFS_DIRHASH		# Improve performance on big directories
options 	MD_ROOT			# MD is a potential root device
options 	NFSCLIENT		# Network Filesystem Client
options 	NFSSERVER		# Network Filesystem Server
options 	NFS_ROOT		# NFS usable as /, requires NFSCLIENT
options 	MSDOSFS			# MSDOS Filesystem
options 	CD9660			# ISO 9660 Filesystem
options 	PROCFS			# Process filesystem (requires PSEUDOFS)
options 	PSEUDOFS		# Pseudo-filesystem framework
options		SCTP			# Stream Transmittion Control Protocol
options 	COMPAT_43TTY		# Compatible with BSD 4.3 [KEEP THIS!]
options 	COMPAT_FREEBSD4		# Compatible with FreeBSD4
options 	COMPAT_FREEBSD5		# Compatible with FreeBSD5
options 	COMPAT_FREEBSD6		# Compatible with FreeBSD6
options 	SCSI_DELAY=5000		# Delay (in ms) before probing SCSI
#options		AUDIT			# Audit Framework
options 	KTRACE			# ktrace(1) support
options 	STACK			# stack(9) support
options 	SYSVSHM			# SYSV-style shared memory
options 	SYSVMSG			# SYSV-style message queues
options 	SYSVSEM			# SYSV-style semaphores
options 	_KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
options 	AHC_REG_PRETTY_PRINT	# Print register bitfields in debug
					# output.  Adds ~128k to driver.
options 	AHD_REG_PRETTY_PRINT	# Print register bitfields in debug
					# output.  Adds ~215k to driver.
options 	ADAPTIVE_GIANT		# Giant mutex is adaptive.
options 	STOP_NMI		# Stop CPUs using NMI instead of IPI

options         HZ=1000

options         SHMMAXPGS=131072
options         SEMMNI=128
options         SEMMNS=512
options         SEMUME=100
options         SEMMNU=256

# Bus support.
device		eisa
device		pci

# ATA and ATAPI devices
device		ata
device		atadisk		# ATA disk drives
options 	ATA_STATIC_ID	# Static device numbering

# atkbdc0 controls both the keyboard and the PS/2 mouse
device		atkbdc		# AT keyboard controller
device		atkbd		# AT keyboard
device		psm		# PS/2 mouse

device		vga		# VGA video card driver

device		splash		# Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device		sc

device		agp		# support several AGP chipsets

# Serial (COM) ports
device		sio		# 8250, 16[45]50 based serial ports
device		uart		# Generic UART driver

device		em		# Intel Gigabit Ethernet

# Pseudo devices.
device		loop		# Network loopback
device		random		# Entropy device
device		ether		# Ethernet support
device		ppp		# Kernel PPP
device		tun		# Packet tunnel.
device		pty		# Pseudo-ttys (telnet etc)
device		md		# Memory "disks"

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device		bpf		# Berkeley packet filter

# Crypto Support
device		crypto
device		cryptodev

 -- ifconfig --

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 00:1b:21:26:13:f2
	inet6 fe80::21b:21ff:fe26:13f2%em0 prefixlen 64 scopeid 0x1 
	inet netmask 0xffffff00 broadcast
	media: Ethernet 100baseTX <full-duplex> (100baseTX <half-duplex>)
	status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
	inet netmask 0xff000000 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet netmask 0xffffffff 
	inet6 2001:470:1f07:208::beef:cafe prefixlen 64 
gre0: flags=9051<UP,POINTOPOINT,RUNNING,LINK0,MULTICAST> metric 0 mtu 1476
	tunnel inet -->
	inet6 fe80::21b:21ff:fe26:13f2%gre0 prefixlen 64 scopeid 0x3 
	inet --> netmask 0xfffffffc 
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1280
	tunnel inet -->
	inet6 fe80::21b:21ff:fe26:13f2%gif0 prefixlen 64 scopeid 0x4 
	inet6 2001:470:1f06:208::2 --> 2001:470:1f06:208::1 prefixlen 128 
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 00:bd:de:1c:01:00
	inet6 fe80::2bd:deff:fe1c:100%tap0 prefixlen 64 scopeid 0x5 
	inet netmask 0xffffff00 broadcast
	inet6 2001:470:89e1::1 prefixlen 112 
	Opened by PID 1267
tun10: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	inet6 fe80::21b:21ff:fe26:13f2%tun10 prefixlen 64 scopeid 0x6 
	inet --> netmask 0xffffffff 
	Opened by PID 1300

Basically, the box will run fine for about 36 hours, after which it will panic.
Failing to allocate kmem:
 Panic String: kmem_malloc(4096): kmem_map too small: 335544320 total allocated

When you take a look at netstat -m, you'll see something like this:
68264/226/68490 mbufs in use (current/cache/total)
256/134/390/25600 mbuf clusters in use (current/cache/total/max)
256/128 mbuf+clusters out of packet secondary zone in use (current/cache)
0/39/39/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
17578K/480K/18058K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/35/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
143 requests for I/O initiated by sendfile
0 calls to protocol drain routines

the current/total mbuf numbers will continually grow until they reach over a
million.  bytes allocated to network will also grow (current/total) until it
reaches over 400MB (I increased kmem_size to stave off the inevitable a little

Code like the following:
while true; do
  OLDMBUFS="`vmstat -z | grep mbuf: | awk -F, '{print $3}'`"
  sleep 1800

  NEWMBUFS="`vmstat -z | grep mbuf: | awk -F, '{print $3}'`"


  date >> ${LOG}
  echo "New mbufs since last check: ${DIFF}" >> ${LOG}

Tue Dec 23 18:46:24 EST 2008
New mbufs since last check: 24064
Tue Dec 23 19:16:24 EST 2008
New mbufs since last check: 23775
Tue Dec 23 19:46:25 EST 2008
New mbufs since last check: 23567
Tue Dec 23 20:16:25 EST 2008
New mbufs since last check: 23322

I've tried both fxp0 and em0 with the same result.  I've tried swapped everything
but the hard drive as well.  This problem was occurring in both 7.0-RELEASE-p6
and 7.1-RC1-2.
There's a lot going on on this box as far as networking features being used..
I'm using IPv4, IPv6, if_gif, if_tap, if_tun, etc..  So it's hard to narrow it
down to one thing.  The latest thing I had done was to add an AAAA record for
our primary domain.  However, the IPv6 traffic coming in is fairly light.
No idea.  But I'd love it if there were a way that I could remotely dig into
these mbufs and find out what traffic is triggering the problem.  I have 
vmcore files lying around that I could use.


More information about the freebsd-bugs mailing list