FreeBSD 4.9 losing mbufs!!!

Robert Watson rwatson at FreeBSD.org
Wed Apr 26 17:06:48 UTC 2006


On Wed, 26 Apr 2006, Stephen Clark wrote:

>> Sorry not to have caught this thread earlier; I've been on travel for the 
>> last few weeks.  My general suggestion would be to try to narrow the code 
>> paths traversed to try to eliminate as much code as possible from the 
>> search.  It sounds like you've done that pretty effectively :-).
>> 
>> Typically, memory leaks occur in edge error cases, where the memory is not 
>> properly released, or ownership is unclear.  My suggestion would be to add 
>> counters (or look at existing counters where already present) and see if 
>> there's an error case being triggered in about the same quantity that mbuf 
>> leakage is occuring.  Chances are, there's an error being returned and a 
>> missing m_freem().
>> 
>> Based on your comments above, I might also pay attention to the routing 
>> socket path -- the rate of leak could correspond to the routing daemons 
>> talking to the network stack, rather than the rate of traffic.  For 
>> example, it could be that one of the routing messages is handled improperly 
>> resulting in a leak.
>> 
>> Unfortunately, tracking down memory leaks can be quite difficult, and tends 
>> to require a combination of dogged persistence and luck...
>
> Good news and bad news.
>
> I managed to get enough of our system running on 6.x stable to test and it 
> does not appear to lose mbufs. Bad news my ipsec transfer rate dropped from 
> 54mbits/sec to 39mbits/sec. We need to be able to handle a t3 (45mbits/sec). 
> Any ideas as to why this drop off in 6.x?

Well, that's good about the good news, but not so great about the bad news. 
Are you using IPv6?  If not, could you look at how FAST_IPSEC performs instead 
of the default IPSEC package, if you're not already doing so?  The KAME IPSEC 
implementation is not multi-processor compatible, so when IPSEC support is 
compiled into the kernel, execution of the network stack and related 
components is limited to a single CPU (you'll see a warning about this at boot 
if this is the case).  This is something we hope to fix in a future release, 
but in the mean time FAST_IPSEC offers improved performance and SMP 
scalability, as well as support for hardware crypto acceleration.  The 
downside to FAST_IPSEC is that it doesn't currently support IPv6, which is 
also something we'd like to fix in the future.

Aside from the above, there are a number of other things we can look at to 
decide what the source of the performance problem is.  First, I'd encourage 
you to make sure any system debugging features, such as INVARIANTS, 
INVARIANT_SUPPORT, and WITNESS, are disabled.  Next, I would recommend 
generating a high level of your load on the system, and using systat -vmstat 1 
and top -S to grab some information about it in the steady state.  I recommend 
letting both run for a couple of minutes, then grabbing the output from both 
of them.  This will give us an idea of where CPU usage is going in the kernel, 
which is where I assume most of your workload ends up being handled.

Thanks,

Robert N M Watson


More information about the freebsd-stable mailing list