kern/165903: mbuf leak
Chris Forgeron
cforgeron at acsi.ca
Sat Apr 13 23:03:40 UTC 2013
To follow-up:
I installed 9.1-RELEASE last night, then went and cvsup'ed to the latest that RELENG_9 would give me.
Everything is completely stock, I have not modified any config files other than filling out the setup questions (hostname, em0 set to DHCP, added a base user so I can ssh to it)
Mu current uname -a is: FreeBSD test 9.1-STABLE FreeBSD 9.1-STABLE #0: Sat Apr 13 00:29:24 ADT 2013 root at test:/usr/obj/usr/src/sys/GENERIC amd64
I am still having the same mbuf drain problem, that doesn't exist on 9.0-STABLE of July 2012.
It's only been 2 hours since my last reboot, and you can already see the drain starting. I expect this box will be unresponsive on the network sometime tomorrow.
All that has been running is a ssh connection that wasn't moving any traffic.
Here is a dump of the requested logs. Please note that I give the stats at boot, then ~2 hours later.
Please let me know how I can help this move forward.
-------------------------------------------------
After Boot
Sat Apr 13 17:41:04 ADT 2013
root at test:/usr/home/aatech # vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 89, 13, 89, 0, 0
UMA Zones: 640, 0, 89, 1, 89, 0, 0
UMA Slabs: 568, 0, 3817, 5, 6208, 0, 0
UMA RCntSlabs: 568, 0, 273, 0, 273, 0, 0
UMA Hash: 256, 0, 3, 12, 4, 0, 0
16 Bucket: 152, 0, 64, 11, 64, 0, 0
32 Bucket: 280, 0, 51, 5, 51, 0, 0
64 Bucket: 536, 0, 34, 1, 34, 62, 0
128 Bucket: 1048, 0, 41, 1, 41, 676, 0
VM OBJECT: 232, 0, 977, 31, 10100, 0, 0
MAP: 232, 0, 7, 25, 7, 0, 0
KMAP ENTRY: 120, 150257, 33, 246, 8450, 0, 0
MAP ENTRY: 120, 0, 753, 115, 23051, 0, 0
fakepg: 120, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 323, 4, 323, 0, 0
16: 16, 0, 2486, 202, 38914, 0, 0
32: 32, 0, 2990, 444, 16915, 0, 0
64: 64, 0, 7243, 261, 37658, 0, 0
128: 128, 0, 8311, 186, 12364, 0, 0
256: 256, 0, 465, 75, 3012, 0, 0
512: 512, 0, 613, 66, 3743, 0, 0
1024: 1024, 0, 34, 162, 4785, 0, 0
2048: 2048, 0, 5129, 141, 5692, 0, 0
4096: 4096, 0, 421, 34, 6083, 0, 0
Files: 80, 0, 78, 102, 3712, 0, 0
rl_entry: 40, 0, 29, 223, 29, 0, 0
TURNSTILE: 136, 0, 115, 45, 115, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1184, 0, 44, 28, 893, 0, 0
THREAD: 1160, 0, 98, 16, 98, 0, 0
SLEEPQUEUE: 80, 0, 115, 30, 115, 0, 0
VMSPACE: 392, 0, 26, 24, 876, 0, 0
cpuset: 72, 0, 51, 49, 51, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 288, 224, 1462, 0, 0
mbuf: 256, 0, 2, 266, 380, 0, 0
mbuf_cluster: 2048, 25600, 512, 16, 512, 0, 0
mbuf_jumbo_page: 4096, 12800, 0, 9, 6, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
g_bio: 232, 0, 0, 144, 3498, 0, 0
ttyinq: 160, 0, 180, 60, 765, 0, 0
ttyoutq: 256, 0, 95, 25, 407, 0, 0
ata_request: 328, 0, 0, 48, 86, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0
FPU_save_area: 832, 0, 0, 0, 0, 0, 0
VNODE: 504, 0, 490, 30, 516, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 36, 7095, 0, 0
S VFS Cache: 108, 0, 475, 53, 1000, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 0, 0, 0, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
NCLNODE: 568, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 23, 13, 23, 0, 0
pipe: 728, 0, 2, 28, 460, 0, 0
Mountpoints: 824, 0, 2, 6, 2, 0, 0
ksiginfo: 112, 0, 57, 999, 63, 0, 0
itimer: 344, 0, 1, 21, 1, 0, 0
KNOTE: 128, 0, 0, 87, 40, 0, 0
socket: 680, 25602, 23, 13, 283, 0, 0
ipq: 56, 819, 0, 0, 0, 0, 0
udp_inpcb: 392, 25600, 8, 22, 222, 0, 0
udpcb: 16, 25704, 8, 328, 222, 0, 0
tcp_inpcb: 392, 25600, 4, 26, 10, 0, 0
tcpcb: 976, 25600, 4, 8, 10, 0, 0
tcptw: 72, 5150, 0, 0, 0, 0, 0
syncache: 152, 15375, 0, 50, 1, 0, 0
hostcache: 136, 15372, 0, 0, 0, 0, 0
tcpreass: 40, 1680, 0, 0, 0, 0, 0
sackhole: 32, 0, 0, 0, 0, 0, 0
sctp_ep: 1376, 25600, 0, 0, 0, 0, 0
sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 216, 3, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 25600, 1, 19, 1, 0, 0
unpcb: 240, 25600, 8, 40, 40, 0, 0
rtentry: 200, 0, 13, 44, 14, 0, 0
selfd: 56, 0, 38, 151, 2435, 0, 0
SWAPMETA: 288, 502372, 0, 0, 0, 0, 0
FFS inode: 168, 0, 453, 53, 477, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 453, 27, 477, 0, 0
root at test:/usr/home/aatech # netstat -Q
Configuration:
Setting Current Limit
Thread count 1 1
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs disabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 256 flow default ---
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 7 256 source default ---
ether 9 256 source direct ---
ip6 10 256 flow default ---
Workstreams:
WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled
0 0 ip 0 0 6 0 0 0 6
0 0 igmp 0 0 0 0 0 0 0
0 0 rtsock 0 2 0 0 0 16 16
0 0 arp 0 0 1 0 0 0 1
0 0 ether 0 0 7 0 0 0 7
0 0 ip6 0 0 0 0 0 0 0
root at test:/usr/home/aatech # netstat -n -x
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address R-MBUF S-MBUF R-CLUS S-CLUS R-HIWA S-HIWA R-LOWA S-LOWA
R-BCNT S-BCNT R-BMAX S-BMAX rexmt persist keep 2msl delack rcvtime
tcp4 0 52 192.168.9.224.22 192.168.9.108.61563 0 1 0 0 65700 33580 1 2048
0 256 525600 268640 0.37 0.00 7199.90 0.00 0.00 0.10
udp4 0 0 127.0.0.1.123 *.* 0 0 0 0 42080 9216 1 2048
0 0 336640 73728
udp6 0 0 fe80:4::1.123 *.* 0 0 0 0 42080 9216 1 2048
0 0 336640 73728
udp6 0 0 ::1.123 *.* 0 0 0 0 42080 9216 1 2048
0 0 336640 73728
udp4 0 0 192.168.9.224.123 *.* 0 0 0 0 42080 9216 1 2048
0 0 336640 73728
Active UNIX domain sockets
Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr
fffffe00064ab690 stream 0 0 0 fffffe00064ab5a0 0 0
fffffe00064ab5a0 stream 0 0 0 fffffe00064ab690 0 0
fffffe00064a00f0 stream 0 0 fffffe00063e8bd0 0 0 0 /var/run/devd.pipe
fffffe00064aa5a0 dgram 0 0 0 fffffe00064aa960 0 0
fffffe00064ab0f0 dgram 0 0 0 fffffe00064aa870 0 fffffe00064ab1e0
fffffe00064ab1e0 dgram 0 0 0 fffffe00064aa870 0 0
fffffe00064aa870 dgram 0 0 fffffe0006531bd0 0 fffffe00064ab0f0 0 /var/run/logpriv
fffffe00064aa960 dgram 0 0 fffffe0006531dc8 0 fffffe00064aa5a0 0 /var/run/log
root at test:/usr/home/aatech # netstat -m
298/737/1035 mbufs in use (current/cache/total)
296/360/656/25600 mbuf clusters in use (current/cache/total/max)
296/344 mbuf+clusters out of packet secondary zone in use (current/cache)
0/9/9/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
666K/940K/1606K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
----------------------------------------------------
After ~2 hours
root at test:/usr/home/aatech # date
Sat Apr 13 19:32:34 ADT 2013
root at test:/usr/home/aatech # netstat -m
1616/694/2310 mbufs in use (current/cache/total)
1614/322/1936/25600 mbuf clusters in use (current/cache/total/max)
1614/306 mbuf+clusters out of packet secondary zone in use (current/cache)
0/9/9/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
3632K/853K/4485K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
root at test:/usr/home/aatech # vmstat -z
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 89, 13, 89, 0, 0
UMA Zones: 640, 0, 89, 1, 89, 0, 0
UMA Slabs: 568, 0, 3851, 6, 6244, 0, 0
UMA RCntSlabs: 568, 0, 977, 3, 977, 0, 0
UMA Hash: 256, 0, 3, 12, 4, 0, 0
16 Bucket: 152, 0, 65, 10, 65, 0, 0
32 Bucket: 280, 0, 51, 5, 51, 0, 0
64 Bucket: 536, 0, 40, 2, 40, 62, 0
128 Bucket: 1048, 0, 43, 2, 43, 676, 0
VM OBJECT: 232, 0, 1001, 87, 13050, 0, 0
MAP: 232, 0, 7, 25, 7, 0, 0
KMAP ENTRY: 120, 150257, 33, 246, 8454, 0, 0
MAP ENTRY: 120, 0, 753, 146, 29632, 0, 0
fakepg: 120, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 323, 4, 323, 0, 0
16: 16, 0, 2486, 202, 40587, 0, 0
32: 32, 0, 2990, 444, 17187, 0, 0
64: 64, 0, 7260, 300, 41779, 0, 0
128: 128, 0, 8367, 275, 13151, 0, 0
256: 256, 0, 469, 71, 4864, 0, 0
512: 512, 0, 637, 91, 4628, 0, 0
1024: 1024, 0, 34, 162, 5307, 0, 0
2048: 2048, 0, 5129, 141, 5694, 0, 0
4096: 4096, 0, 421, 66, 6369, 0, 0
Files: 80, 0, 78, 147, 5507, 0, 0
rl_entry: 40, 0, 43, 209, 43, 0, 0
TURNSTILE: 136, 0, 133, 27, 133, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1184, 0, 44, 43, 1158, 0, 0
THREAD: 1160, 0, 116, 16, 116, 0, 0
SLEEPQUEUE: 80, 0, 133, 41, 133, 0, 0
VMSPACE: 392, 0, 26, 24, 1076, 0, 0
cpuset: 72, 0, 51, 49, 51, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 1622, 298, 31168, 0, 0
mbuf: 256, 0, 2, 388, 5267, 0, 0
mbuf_cluster: 2048, 25600, 1920, 16, 1920, 0, 0
mbuf_jumbo_page: 4096, 12800, 0, 9, 6, 0, 0
mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0
mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
g_bio: 232, 0, 0, 144, 5175, 0, 0
ttyinq: 160, 0, 180, 60, 765, 0, 0
ttyoutq: 256, 0, 95, 25, 407, 0, 0
ata_request: 328, 0, 0, 48, 2315, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, 0
FPU_save_area: 832, 0, 0, 0, 0, 0, 0
VNODE: 504, 0, 497, 23, 537, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 36, 13140, 0, 0
S VFS Cache: 108, 0, 471, 123, 1525, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 0, 0, 0, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
NCLNODE: 568, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 23, 13, 23, 0, 0
pipe: 728, 0, 2, 28, 534, 0, 0
Mountpoints: 824, 0, 2, 6, 2, 0, 0
ksiginfo: 112, 0, 75, 981, 311, 0, 0
itimer: 344, 0, 1, 21, 1, 0, 0
KNOTE: 128, 0, 0, 87, 40, 0, 0
socket: 680, 25602, 23, 25, 443, 0, 0
ipq: 56, 819, 0, 0, 0, 0, 0
udp_inpcb: 392, 25600, 8, 22, 336, 0, 0
udpcb: 16, 25704, 8, 328, 336, 0, 0
tcp_inpcb: 392, 25600, 4, 26, 14, 0, 0
tcpcb: 976, 25600, 4, 8, 14, 0, 0
tcptw: 72, 5150, 0, 100, 2, 0, 0
syncache: 152, 15375, 0, 50, 3, 0, 0
hostcache: 136, 15372, 0, 0, 0, 0, 0
tcpreass: 40, 1680, 0, 0, 0, 0, 0
sackhole: 32, 0, 0, 0, 0, 0, 0
sctp_ep: 1376, 25600, 0, 0, 0, 0, 0
sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 216, 3, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 25600, 1, 19, 1, 0, 0
unpcb: 240, 25600, 8, 40, 82, 0, 0
rtentry: 200, 0, 13, 44, 14, 0, 0
selfd: 56, 0, 42, 147, 64963, 0, 0
SWAPMETA: 288, 502372, 0, 0, 0, 0, 0
FFS inode: 168, 0, 459, 47, 496, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 459, 21, 496, 0, 0
root at test:/usr/home/aatech #
root at test:/usr/home/aatech # netstat -Q
Configuration:
Setting Current Limit
Thread count 1 1
Default queue limit 256 10240
Dispatch policy direct n/a
Threads bound to CPUs disabled n/a
Protocols:
Name Proto QLimit Policy Dispatch Flags
ip 1 256 flow default ---
igmp 2 256 source default ---
rtsock 3 256 source default ---
arp 7 256 source default ---
ether 9 256 source direct ---
ip6 10 256 flow default ---
Workstreams:
WSID CPU Name Len WMark Disp'd HDisp'd QDrops Queued Handled
0 0 ip 0 0 166 0 0 0 166
0 0 igmp 0 0 0 0 0 0 0
0 0 rtsock 0 2 0 0 0 16 16
0 0 arp 0 0 63 0 0 0 63
0 0 ether 0 0 229 0 0 0 229
0 0 ip6 0 0 0 0 0 0 0
root at test:/usr/home/aatech #
root at test:/usr/home/aatech # netstat -n -x
Active Internet connections
Proto Recv-Q Send-Q Local Address Foreign Address R-MBUF S-MBUF R-CLUS S-CLUS R-HIWA S-HIWA R-LOWA S-LOWA
R-BCNT S-BCNT R-BMAX S-BMAX rexmt persist keep 2msl delack rcvtime
tcp4 0 52 192.168.9.224.22 192.168.9.108.61563 0 1 0 0 65700 33580 1 2048
0 256 525600 268640 0.41 0.00 7199.92 0.00 0.00 0.08
udp4 0 0 127.0.0.1.123 *.* 0 0 0 0 42080 9216 1 2048
0 0 336640 73728
udp6 0 0 fe80:4::1.123 *.* 0 0 0 0 42080 9216 1 2048
0 0 336640 73728
udp6 0 0 ::1.123 *.* 0 0 0 0 42080 9216 1 2048
0 0 336640 73728
udp4 0 0 192.168.9.224.123 *.* 0 0 0 0 42080 9216 1 2048
0 0 336640 73728
Active UNIX domain sockets
Address Type Recv-Q Send-Q Inode Conn Refs Nextref Addr
fffffe00064ab690 stream 0 0 0 fffffe00064ab5a0 0 0
fffffe00064ab5a0 stream 0 0 0 fffffe00064ab690 0 0
fffffe00064a00f0 stream 0 0 fffffe00063e8bd0 0 0 0 /var/run/devd.pipe
fffffe00064aa5a0 dgram 0 0 0 fffffe00064aa960 0 0
fffffe00064ab0f0 dgram 0 0 0 fffffe00064aa870 0 fffffe00064ab1e0
fffffe00064ab1e0 dgram 0 0 0 fffffe00064aa870 0 0
fffffe00064aa870 dgram 0 0 fffffe0006531bd0 0 fffffe00064ab0f0 0 /var/run/logpriv
fffffe00064aa960 dgram 0 0 fffffe0006531dc8 0 fffffe00064aa5a0 0 /var/run/log
root at test:/usr/home/aatech #
More information about the freebsd-stable
mailing list