Sudden mbuf demand increase and shortage under the load
alfred at freebsd.org
Tue Feb 16 00:26:16 UTC 2010
* Maxim Sobolev <sobomax at FreeBSD.org> [100215 04:49] wrote:
> Our company have a FreeBSD based product that consists of the numerous
> interconnected processes and it does some high-PPS UDP processing
> (30-50K PPS is not uncommon). We are seeing some strange periodic
> failures under the load in several such systems, which usually evidences
> itself in IPC (even through unix domain sockets) suddenly either
> breaking down or pausing and restoring only some time later (like 5-10
> minutes). The only sign of failure I managed to find was the increase of
> the "requests for mbufs denied" in the netstat -m and number of total
> mbuf clusters (nmbclusters) raising up to the limit.
Can you run a process to dump sysctl -a every second or so
and mark the time when you did it?
Other monitoring things would probably be helpful as well (netstat -m)
in a timed log format. vmstat -i? (interrupts storm?)
Perhaps ps output (showing interrupt threads, etc) would be good toknow
perhaps some ithreads went off into the weeds...
Any console messages of note?
A few people have suggested that there may be too many packets
on the outgoing interface, I think there should be a limit to the
number of packets queued for outgoing and probably counters to
show how many were dropped due to overflow of the outgoing queue.
You should be able to check these counters to see what is going
on. If the driver is broken and never drops outgoing packets when
the card's queue is full, then those counters should be 0.
I hope this helps.
More information about the freebsd-stable