8.0-RELEASE-p3: 4k jumbo mbuf cluster exhaustion

[Pyun YongHyeon]

On Sun, Aug 22, 2010 at 05:40:30PM +0800, Adrian Chadd wrote:
> I disabled tso, tx chksum and rx chksum. This fixed the 4k jumbo
> allocation growth.
> 

I recall there was SIOCSIFCAP ioctl handling bug in bce(4) on 8.0 so
it might also disable IFCAP_TSO4/IFCAP_TXCSUM/IFCAP_RXCSUM when yo
disabled RX checksum offloading. But I can't explain how checksum
offloading could be related with the growth of 4k jumbo buffers.
> 
> Turning on tso on a live proxy didn't affect jumbo allocations.
> Turning on txcsum caused jumbo allocations to begin growing again.
> DIsabling txcsum again caused jumbo allocations to stop increasing,
> but it doesn't seem to be decreasing back to the steady state (~ 8k.)
> Turning on rxcsum didn't affect jumbo allocations.
> 
> So it seems txcsum is the culprit here.
> 

There was a lot of changes in bce(4) since 8.0-RELEASE. I vaguely
guess your issue could be related with header split feature of
bce(4) which was now disabled. Are you using jumbo
frame/ZERO_COPY_SOCKETS with bce(4)?

> 
> 
> Adrian
> 
> On 22 August 2010 16:11, Adrian Chadd <adrian.chadd at gmail.com> wrote:
> > Hi,
> >
> > I've got a Squid/Lusca server on 8.0-RELEASE-p3 which is exhibiting
> > some very strange behaviour.
> >
> > After a few minutes uptime, the 4k mbuf cluster zone fills up and
> > Squid/Lusca spends almost all of it's time sleeping in "keglimit".
> >
> > I've bumped kern.ipc.nmbclusters to 262144 and kern.ipc.jumbop to
> > 32768 but the system will slowly crawl towards filling that zone.
> >
> > The box has a bce on-board NIC and is using ipfw to handle redirecting
> > traffic to/from the box for transparent TCP interception. It's
> > handling around ~30,000 concurrent connections at the moment.
> >
> > I have other very busy proxies on FreeBSD-7.x pushing a few hundred
> > megabits without any issues. This box falls over after ~ 20 mbit.
> >
> > If I bypass redirection and/or kill squid, the 4k cluster count drops
> > back down to < 500 and stays there.
> >
> > Does anyone have any ideas on where to begin debugging this?
> >
> > Thanks,
> >
> >
> > Adrian
> >