VIMAGE: Freed UMA keg was not empty

Thierry Herbelot thierry.herbelot at free.fr
Wed Nov 17 18:36:34 UTC 2010


Thierry Herbelot <thierry.herbelot at free.fr> a écrit
> "Bjoern A. Zeeb" <bzeeb-lists at lists.zabbadoz.net> a écrit
> 
> > On Wed, 17 Nov 2010, Thierry Herbelot wrote:
> > 
> > Hi,
> > 
> > first of all freebsd-virtualization@ is the better list for this; Cc:ed.
> 
> (in fact, I did not know where else to send this message : -net, ... ?
> thanks for the CC)
> 
> > > We are using FreeBSD + VIMAGE at work, and we have seen an annoying
> > > problem : there seems to be a memory leak in the kernel, which
> > > eventually causes a panic.
> > > 
> > > (yes, we have seen the following message : "WARNING: VIMAGE
> > > (virtualized network stack) is a highly experimental feature.")
> > > 
> > > Configuring a network interface in a jail with vnet enabled, then
> > > removing that jail causes these messages. In dmesg:
> > > 
> > > [...]
> > > Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
> > > Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
> > > 
> > > The issue happens in a GENERIC FreeBSD 8.1 kernel, with VIMAGE enabled
> > > (with attached VIMAGE kernel config file).
> > > 
> > > The following commands reproduce the bug:
> > > 
> > > jail -l -u root -c path=/ name=foo persist vnet &&
> > > jexec foo ifconfig lo0 127.0.0.1/8 &&
> > > jail -r foo
> > > 
> > > Running it too many times exhausts kernel memory and crashes the
> > > machine.
> > > 
> > > The probleme is seen on 8.1-release kernel, 8-stable from SVN and SVN
> > > -head.
> > 
> > The problem has been present since day 1 and is still present up to
> > HEAD.  This is about type stability and teardown.  So far we are no
> > able to actually free TCP (and UDP in SVN) states as they might still
> > be accesses after "free".  It's a general problem in the network stack
> > and has been implemented as a measure to circumvent panics in those
> > cases.
> > 
> > I am not sure if that's actually what's biting you wrt to memory or
> > if it's something else.  Unfortunately boot logs and kernel configs
> > don't help much; enable ddb and get show uma and show malloc reports
> > after the crash (or watch vmstat -z and vmstat -m) after every 50 jail
> > restarts.  I might have some patches for a couple of things but cannot
> > (yet) help with the additional (duplicate) UMA zones showing up at
> > each iteration (for the -z case).
> 
> The context for our problem is that VIMAGE jails are mant to be used "in
> production" for automated tests, thus are likely to be multiple on one
> physical host, and are likely to be started and stopped according to our
> needs.
> 
> We have tried more "classical" virtualization solutions (qemu, kvm, ... ,
> normal jails with individual FIBs, ...) but only FreeBSD with VIMAGE seems
> to be the correct answer. (for other reasons, an i386 version of the
> kernel must be used)
> 
> The point of my original email was to simplify the configuration for a test
> setup : any machine with the attached configuration file and the above
> sequence of commands creating and deleting jails, running in a loop, will
> eventually panic.
> 
> Anyway, I will follow up with the logs you mentioned.

As promised, here are the full logs (in attachment)

This is a serial console log showing the command loop that triggers the bug on 
a debug kernel and ensuing DDB session.

the obvious problem line is :
          routetbl         2684       303890K         3469

(further tests showed an increase of the routetbl malloc zone by 4MBytes for 
each vnet jail creation/destruction cycle)

	Cheers

	Thierry

> 
> 	Cheers (and thanks for the quick feedback)
> 
> 	Thierry Herbelot
> 
> > /bz
> 
> _______________________________________________
> freebsd-current at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe at freebsd.org"
-------------- next part --------------
[root at waldorf ~]# while jail -l -u root -c path=/ name=foo persist vnet && jexe
c foo ifconfig lo0 127.0.0.1/8 && jail -r foo ; do continue ; done
lock order reversal:
 1st 0xc0bed490 allprison (allprison) @ kern/kern_jail.c:914
 2nd 0xc0d5c56c vnet_sysinit_sxlock (vnet_sysinit_sxlock) @ net/vnet.c:618
KDB: stack backtrace:
db_trace_self_wrapper(c0ab1220,e9867a08,c05fc285,c05ec9db,c0ab40dc,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c05ec9db,c0ab40dc,c6d2d138,c6d30c88,e9867a64,...) at kdb_backtrace+0x29
_witness_debugger(c0ab40dc,c0d5c56c,c0abf638,c6d30c88,c0abf794,...) at _witness_debugger+0x25
witness_checkorder(c0d5c56c,1,c0abf78b,26a,0,...) at witness_checkorder+0x839
_sx_slock(c0d5c56c,0,c0abf78b,26a,0,...) at _sx_slock+0x85
vnet_sysinit(c7acf000,c0bb5d00,6180,0,c0b80888,...) at vnet_sysinit+0x2b
vnet_alloc(c6f73028,c0aa9de6,0,10,c0aa5dc0,...) at vnet_alloc+0x9e
kern_jail_set(c77ed000,c767b380,1,c767b380,280930fc,...) at kern_jail_set+0x179a
jail_set(c77ed000,e9867cf8,c0ae5dd0,c0ab4ea5,c77e87f8,...) at jail_set+0x50
syscall(e9867d38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (507, FreeBSD ELF32, jail_set), eip = 0x280eae1b, esp = 0xbf7feb7c, ebp = 0xbf7fec38 ---
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
Freed UMA keg was not empty (203 items).  Lost 1 pages of memory.
Freed UMA keg was not empty (36 items).  Lost 2 pages of memory.
panic: kmem_malloc(131072): kmem_map too small: 335409152 total allocated
cpuid = 0
KDB: enter: panic
[thread pid 2129 tid 100072 ]
Stopped at      kdb_enter+0x3a: movl    $0,kdb_why
db> bt
Tracing pid 2129 tid 100072 td 0xc771fc80
kdb_enter(c0aadf11,c0aadf11,c0ad7eba,e95fa94c,0,...) at kdb_enter+0x3a
panic(c0ad7eba,20000,13fdf000,c0ad7eb4,7d0,...) at panic+0x136
kmem_malloc(c149008c,20000,102,e95fa9d8,c082951d,...) at kmem_malloc+0x280
page_alloc(0,20000,e95fa9cb,102,0,...) at page_alloc+0x27
uma_large_malloc(20000,102,102,2,1,...) at uma_large_malloc+0x4d
malloc(20000,c0b90240,102,c0585588,20000,...) at malloc+0x118
flowtable_alloc(c0ac790e,8000,2,0,0,...) at flowtable_alloc+0xee
ip_init(40,c7629dc0,0,e95faa80,c0617807,...) at ip_init+0x330
protosw_init(40,c0b8e754,c0b941b4,c7629dc0,e95faa8c,...) at protosw_init+0x139
domain_init(c0b93f20,e95faaa8,c068bdde,c0b93f20,0,...) at domain_init+0x27
vnet_domain_init(c0b93f20,0,c0abf78b,26a,0,...) at vnet_domain_init+0x11
vnet_sysinit(c718c000,c0bb5d00,6180,0,c0b80888,...) at vnet_sysinit+0x3e
vnet_alloc(c877e028,c0aa9de6,0,10,c0aa5dc0,...) at vnet_alloc+0x9e
kern_jail_set(c771fc80,c7676e80,1,c7676e80,280930fc,...) at kern_jail_set+0x179a
jail_set(c771fc80,e95facf8,c0ae5dd0,c0ab4ea5,c771ad48,...) at jail_set+0x50
syscall(e95fad38) at syscall+0x220
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (507, FreeBSD ELF32, jail_set), eip = 0x280eae1b, esp = 0xbf7feb7c, ebp = 0xbf7fec38 ---
db> show uma
              Zone     Size     Used     Free     Requests
               ipq       32        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
           rtentry      108        0       72            6
             ripcb      220        0        0            0
       FFS2 dinode      256      397       83          420
       FFS1 dinode      128        0        0            0
         FFS inode      116      397       98          420
       Mountpoints      644        3       15            3
          SWAPMETA      276        0        0            0
           ip6flow       64        0        0            0
           ip4flow       40        0        0            0
             selfd       28       13      622          953
           rtentry      108        9       99            9
             unpcb      172        4       88           41
             ripcb      220        0        0            0
   sctp_asconf_ack       24        0        0            0
       sctp_asconf       24        0        0            0
sctp_stream_msg_out       64        0        0            0
        sctp_readq       76        0        0            0
        sctp_chunk       92        0        0            0
        sctp_raddr      432        0        0            0
        sctp_laddr       24        0      145            3
         sctp_asoc     1484        0        0            0
           sctp_ep      860        0        0            0
          sackhole       20        0        0            0
          tcpreass       20        0        0            0
         hostcache       76        0        0            0
          syncache      112        0        0            0
             tcptw       52        0        0            0
             tcpcb      632        2       10            2
         tcp_inpcb      220        2       34            2
             udpcb        8        2      404          215
         udp_inpcb      220        2       70          215
               ipq       32        0        0            0
            socket      412        8       28          405
     bridge_rtnode       36       20      283           24
             KNOTE       72        0        0            0
            itimer      220        0        0            0
          ksiginfo       80       56      232          153
              pipe      392        0       90         1453
           NFSNODE      468        0        0            0
          NFSMOUNT      524        0        0            0
           DIRHASH     1024        0       48           26
             NAMEI     1024        0       48        17675
       L VFS Cache      292        0        0            0
       S VFS Cache       72      410      226          917
         VNODEPOLL       60        0        0            0
             VNODE      268      422       68          447
     ata_composite      180        0        0            0
       ata_request      204        0      190         2955
           ttyoutq      256      156       84          420
            ttyinq      152      300      168          810
             g_bio      140        0      224        15483
   mbuf_ext_refcnt        4        0        0            0
    mbuf_jumbo_16k    16384        0        0            0
     mbuf_jumbo_9k     9216        0        0            0
   mbuf_jumbo_page     4096        0        0            0
      mbuf_cluster     2048      512       42          512
              mbuf      256        1      387         3924
       mbuf_packet      256      256      256         1367
      audit_record      816        0        0            0
            cpuset       40        2      366          148
           VMSPACE      236       17      159         2112
        SLEEPQUEUE       48      133      221          133
            THREAD      636      113       19          113
              PROC      680       35       37         2129
        MAC labels       20        0        0            0
           umtx pi       52        0        0            0
         TURNSTILE       72      133       77          133
             Files       56       40      563         8850
              4096     4096     2534       28         7824
              2048     2048      273      109         1052
              1024     1024       68      192         3976
               512      512       92       76         4304
               256      256      904      131        16844
               128      128     2546      484        12434
                64       64     5488      707        49123
                32       32     3077      426        40826
                16       16     3258     1005        42186
           mt_zone     2056      270        0          270
         SG fakepg       76        0        0            0
         DP fakepg       76        0        0            0
              PDPT       32      137      428          137
         MAP ENTRY       72      280      356        51654
        KMAP ENTRY       72       59      471        34398
               MAP      140        7       21            7
         VM OBJECT      136      667      232        27531
        128 Bucket      524       93       12         1366
         64 Bucket      268      206       18          260
         32 Bucket      140       78       34          259
         16 Bucket       76       56       44           82
          UMA Hash      128        4       26            4
     UMA RCntSlabs      544      277        3          277
         UMA Slabs      284     5388       16        18442
         UMA Zones      888      237       19         1770
          UMA Kegs      128      237       33         1770
db> show malloc
              Type        InUse        MemUse     Requests
        CAM periph            2            1K           12
           scsi_cd            0            0K            0
         acpicmbat            0            0K            0
              $PIR            0            0K            0
           acpidev           68            3K           68
           CAM SIM            1            1K            1
       isofs_mount            0            0K            0
        isofs_node            0            0K            0
            isadev            8            1K            8
        ast_driver            0            0K            0
        afd_driver            0            0K            0
          nexusdev            4            1K            4
               msi            3            1K            3
           mptable            0            0K            0
           memdesc            1            4K            1
               MCA            0            0K            0
        acd_driver            1            2K            1
         ar_driver            0            0K            6
         legacydrv            0            0K            0
           io_apic            1            1K            1
         ad_driver            1            1K            1
           ata_pci            0            0K            0
           ata_dma            0            0K            0
       ata_generic            2            2K            2
           scsi_da            0            0K            0
        madt_table            0            0K            0
         CAM queue            3            1K            7
              GEOM           66          181K         5441
            apmdev            2            1K            3
          mpt_user            0            0K            0
     CAM dev queue            1            1K            1
       pfs_vncache            0            0K            0
         pfs_nodes           21            3K           21
      nullfs_mount            0            0K            0
          atkbddev            2            1K            2
       nullfs_hash            1            1K            1
         vm_pgdata            1           64K            1
       nullfs_node            0            0K            0
     msdosfs_mount            0            0K            0
       msdosfs_fat            0            0K            0
           UMAHash            0            0K            0
         ufs_mount            6           93K            6
         ufs_quota            0            0K            0
       ufs_dirhash            9            5K           27
           pagedep            1           64K            1
          inodedep            1          256K            1
            newblk            1            1K            1
         bmsafemap            0            0K            0
       allocdirect            0            0K            0
          indirdep            0            0K            0
        allocindir            0            0K            0
          freefrag            0            0K            0
          freeblks            0            0K            0
          freefile            0            0K            0
            diradd            0            0K            0
             mkdir            0            0K            0
            dirrem            0            0K            0
         newdirblk            0            0K            0
          savedino            0            0K            0
           mactemp            0            0K            0
     audit_trigger            0            0K            0
        audit_pipe            0            0K            0
     audit_pipeent            0            0K            0
 audit_pipe_presel            0            0K            0
     audit_evclass          172            3K          211
         audit_bsm            0            0K            0
        audit_cred            0            0K            0
        audit_data            0            0K            0
        audit_path            0            0K            0
        audit_text            0            0K            0
      audit_gidset            0            0K            0
               rpc            2            5K            2
               NLM            0            0K            0
      nfss_srvsock            0            0K            0
      nfss_srvdesc            0            0K            0
       nfss_daemon            0            0K            0
           NFS FHA            1            1K            1
    nfsclient_lock            0            0K            0
 nfsclient_nlminfo            0            0K            0
     nfsclient_req            0            0K            0
   nfsclient_bigfh            0            0K            0
  nfsclient_diroff            0            0K            0
    nfsclient_hash            0            0K            0
nfsclient_directio            0            0K            0
 nfsclient_srvsock            0            0K            0
               mld           10            2K           83
    msdosfs_fileno            0            0K            0
      msdosfs_node            0            0K            0
       in6_mfilter            0            0K            0
         in6_multi           12            2K          669
      ip6_moptions            0            0K            0
       ip6_msource            0            0K            0
            DEVFSP            0            0K            0
             DEVFS           16            1K           17
        DEVFS_RULE            0            0K            0
          fragment            0            0K            0
          syncache            1           72K           74
            tcplog            0            0K            0
         hostcache            1           16K           74
          sctp_map            0            0K            0
         sctp_stri            0            0K            0
         sctp_stro            0            0K            0
         sctp_aadr            0            0K            0
         sctp_a_it            0            0K          103
         sctp_atcl            0            0K            0
         sctp_atky            0            0K            0
         sctp_athm            0                0
         sctp_athi            0            0K            0
         sctp_stre            0            0K            0
         sctp_cmsg            0            0K            0
         sctp_cpal            0            0K            0
          sctp_vrf            1            1K           74
          sctp_ifa           99           13K          223
          sctp_ifn            2            1K           75
         sctp_timw            0            0K            0
         sctp_mvrf            0            0K            0
         sctp_iter            0            0K          103
        sctp_socko            0            0K            0
 encap_export_host            0            0K            0
        in_mfilter            0            0K            0
          in_multi            2            1K           75
       ip_moptions            0            0K            0
        ip_msource            0            0K            0
            DEVFS2            0            0K            0
            DEVFS3          206           26K          330
            DEVFS1          190           48K          315
              ipid            0            0K            0
              igmp           10            2K           83
         80211scan            0            0K            0
      80211ratectl            0            0K            0
        80211power            0            0K            0
         80211node            0            0K            0
       80211nodeie            0            0K            0
         80211mesh            0            0K            0
          80211com            0            0K            0
          80211dfs            0            0K            0
       80211crypto            0            0K            0
          80211vap            0            0K            0
              vnet            2            1K           75
         vnet_data            2           56K           75
    vnet_data_free            1            1K            1
          routetbl         2684       303890K         3469
     CAM ccb queue            0            0K            0
            ata_da            0            0K            0
           scsi_ch            0            0K            0
            mfibuf            0            0K            0
           md_disk            0            0K            0
        md_sectors            0            0K            0
               LED            2            1K            2
            kbdmux            7           10K            7
              vlan            0            0K            0
               tun            0            0K            0
               amr            0            0K            0
           lltable           23            6K          315
               gif            0            0K            0
            fw_com            0            0K            0
             faith            0            0K            0
            arpcom            8            1K            8
             epair            4            1K            4
             clone            8           32K           81
           ifdescr            0            0K            0
             ifnet           12           11K          158
            ifaddr           93           18K          823
       ether_multi           15            1K          600
               BPF           10            1K           83
  subr_export_host            0            0K            0
             mount           50            3K         2211
       vnodemarker            0            0K          385
           acpisem           15            2K           15
            vnodes            2            1K            2
          vfs_hash            1          256K            1
       export_host            0            0K            0
        cl_savebuf            0            0K            0
          vfscache            1          512K            1
            biobuf            4            8K            6
               acl            0            0K            0
            soname            3            1K           89
               pcb           14           79K          891
            ipsbuf            0            0K            0
             shmfd            1            4K            1
              ksem            1            4K            1
          mbuf_tag            0            0K            1
              accf            0            0K            0
               pts            0            0K            0
               tty           36           18K           38
               shm            1           12K            1
               sem            4            6K            4
               msg            4           25K            4
          ioctlops            0            0K        13640
            select            6            1K            6
               iov            1            1K          745
           Witness            1          104K            1
            iirbuf            0            0K            0
           if_fwip            0            0K            0
           ata_pmp            0            0K            0
               USB           45           68K           46
            Unitno           11            1K          403
         taskqueue           15            1K           15
             stack            0            0K            2
            USBdev           29            7K           29
             USBHC            0            0K            0
            if_fwe            0            0K            0
            sglist            0            0K            0
              sbuf            0            0K         2741
           CAM XPT           12            2K           26
              rman          233           15K          683
          acpitask            1            1K            1
            acpica         3082          163K        65871
           Per-cpu            1            1K            1
              kobj          255          510K          363
            aaccam            0            0K            0
      eventhandler          299           11K          299
           devstat           32           65K           32
               bus         1110           52K         5921
            bus-sc           81          171K         2508
             fwmem            0            0K            0
              SWAP            0            0K            0
          p1003.1b            1            1K            1
              umtx          132            9K          132
           callout            3          768K            3
            sysctl            0            0K         1420
         sysctloid         4004          122K         4124
         sysctltmp            0            0K          535
          firewire            0            0K            0
            plimit           12            3K          355
           uidinfo            2            2K           17
              cred           20            2K        10282
              pgrp           17            2K          472
           session           15            1K           31
              proc            2            8K            2
           subproc          109          163K         2203
           fw_xfer            0            0K            0
               osd            0            0K            0
            aacbuf            0            0K            0
              UART           51           22K           51
          mtx_pool            2            8K            2
      twe_commands            0            0K            0
            module          379           24K          379
      twa_commands            0            0K            0
             cache            0            0K            0
            devbuf         5642        12692K         5728
              temp           59          231K        52869
            ip6opt            0            0K            0
            ip6ndp           11            1K          157
           acpipwr            0            0K            0
              free            0            0K            0
             lockf           14            1K           18
            linker          108            4K          110
       ddb_capture            1           48K            1
            KTRACE          100           13K          100
         ciss_data            0            0K            0
           entropy         1024           64K         1024
            prison            1            2K           74
           ithread           99            8K           99
               PUC            8            2K            8
       Fail Points            0            0K            0
            zombie            0            0K            0
         proc-args           17            1K          771
          ppbusdev            3            1K            3
            kqueue            0            0K            0
              kenv           79            7K           83
          filedesc           40           14K         2572
filedesc_to_leader            0            0K            0
             sigio            1            1K            1
         acpi_perf            4            1K            4
          SCSI SES            0            0K            0
       tty console            0            0K            0
              cdev            9            2K            9
          pci_link           16            2K           16
           SCSI sa            0            0K            0
db>


More information about the freebsd-current mailing list