High usage of mbufs

[Robert Watson]

On Sat, 28 May 2005, Matthew D. Fuller wrote:

> On Sat, May 28, 2005 at 01:17:24PM +1000 I heard the voice of
> Michael VInce, and lo! it spake thus:
>> On one of my web servers I have a really high usage of mbuf clusters in
>> use on a web server that does about 3million hits a day.
>> 4294914731/262144 mbuf clusters in use (current/max)
>
> No it doesn't (have really high usage, that is).  It just has messed up 
> statistics, which is standard on SMP machines; see the archives.

I have two sets of patches that correct this problem:

- One uses atomic operations to synchronize access to the mbuf-related
   counters instead of performing non-atomic updates, which results in
   races.

- One uses per-cpu statistics in the mbuf allocator, and critical sections
   to synchronize updates.

Both encounter a several percent performance hit in packet-intensive 
workloads, so I've been reluctant to commit either.  However, most of the 
information required for mbuf statistics is already gathered by UMA, which 
is now used as the back-end for the mbuf allocator.  As such, the better 
answer is to use the UMA statistics and then see if we still need the mbuf 
layer stats.  I haven't had a chance to investigate this in detail yet, 
but I'd like to get something in before 6.0.

Robert N M Watson