About netstat -m: What is "mbuf+clusters out of packet secondary zone in use" ?

[Robert Watson]

On Mon, 15 Dec 2008, Robert Watson wrote:

> On Mon, 15 Dec 2008, Yony Yossef wrote:
>
>> I'm testing an Ethernet driver on FreeBSD 6.3.
>> 
>> Running netstat -m during an ethernt stress test I see that the 
>> "mbuf+clusters out of packet secondary zone in use" number is growing 
>> gradualy. Problem is it never goes down after I stop the test, so it's 
>> pushing the "mbufs in use" up until the following stress test iterations 
>> reach the OS limit. What does this number mean?

I realized that I didn't quite answer your question here, so to be more 
specific: the FreeBSD mbuf allocator has a number of slab allocator zones it 
can draw from, depending on the type of request.  These are enumerated in the 
vmstat -z output:

robert at fledge:~> vmstat -z | head -1 ; vmstat -z | grep -i mbuf
ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS  FAILURES
mbuf_packet:              256,        0,      262,      222, 202448978,        0
mbuf:                     256,        0,        9,      527, 686757049,        0
mbuf_cluster:            2048,    25600,      484,      304,  4017957,        0
mbuf_jumbo_pagesize:     4096,    12800,        7,      298, 18924376,        0
mbuf_jumbo_9k:           9216,     6400,        0,        0,        0,        0
mbuf_jumbo_16k:         16384,     3200,        0,        0,        0,        0
mbuf_ext_refcnt:            4,        0,        0,      406,  8873247,        0

The 'mbuf' zone allocates simple mbufs from a cache.  The various cluster 
zones allocate cluster storage of various sizes, from the 2k default cluster 
size up to various jumbo sizes used when sending large amounts of data or when 
jumbograms are configured for a network interface that supports them.

The mbuf_packet (mbuf+cluster) zone is a special zone allocating mbufs with 
pre-attached clusters, which was an optimization that seemed to help a lot a 
few years ago.  In particular, you don't have to enter the memory allocator 
twice in order to find an mbuf with a cluster.  There are some downsides, not 
least more complicated book-keeping, and some issues with how to account for 
and manage memory across the various caches.  Notice that all mbuf_packet FREE 
(cached) clusters are billed as used clusters for the mbuf_cluster zone, as 
while UMA knows that mbuf_packet counts as a regular mbuf allocation, it 
doesn't know about the cluster hooked off it; netstat -m adjusts the reported 
cluster allocation for this before printing stats.

Recently, we've been discussing moving to a variable-size mbuf scheme 
supporting large mbuf sizes with embedded storage, which seems to improve 
memory efficient quite a bit.  There are some downsides -- one issue is that 
it somewhat complicates reference-counted cluster use (although not much); 
another is that data is no longer page-aligned which will make page-flipping 
less convenient on the receiver.  Currently, if the hardware supports 
header-spitting, you can imagine the data going fairly easily into 
appropriately sized clusters (especially if the hardware supports LRO 
directly), and then flipping the pages into user memory on receive.  With 
mbufs stuck on the front end of the page, not so much.

Robert N M Watson
Computer Laboratory
University of Cambridge

>
> It seems likely that one of two things is happening:
>
> (1) a leak of mbuf + clusters
> (2) a bug in stats on mbuf + clusters
>
> Could you paste the output of "vmstat -z | grep -i mbuf" into an e-mail? 
> That's the underlying vm stats from the zone used to generate netstat -m 
> output, and might shed more light on things.
>
> Robert N M Watson
> Computer Laboratory
> University of Cambridge
>
>> 
>> 
>> 506391/126009/632400 mbufs in use (current/cache/total)
>> 141035/121109/262144/262144 mbuf clusters in use (current/cache/total/max)
>> 131054/610 mbuf+clusters out of packet secondary zone in use 
>> (current/cache)
>> 
>> 
>> Thanks
>> Yony
>> _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>