Limits on jumbo mbuf cluster allocation

[Andre Oppermann]

On 10.03.2013 07:04, Garrett Wollman wrote:
> <<On Fri, 8 Mar 2013 12:13:28 -0800, Jack Vogel <jfvogel at gmail.com> said:
>
>> Yes, in the past the code was in this form, it should work fine Garrett,
>> just make sure
>> the 4K pool is large enough.
>
> [Andre Oppermann's patch:]
>>> if (adapter->max_frame_size <= 2048)
> adapter-> rx_mbuf_sz = MCLBYTES;
>>> -       else if (adapter->max_frame_size <= 4096)
>>> +       else
> adapter-> rx_mbuf_sz = MJUMPAGESIZE;
>>> -       else if (adapter->max_frame_size <= 9216)
>>> -               adapter->rx_mbuf_sz = MJUM9BYTES;
>>> -       else
>>> -               adapter->rx_mbuf_sz = MJUM16BYTES;
>
> So I tried exactly this, and it certainly worked insofar as only 4k
> clusters were allocated, but NFS performance went down precipitously
> (to fewer than 100 ops/s where normally it would be doing 2000
> ops/s).  I took a tcpdump while it was in this state, which I will try
> to make some sense of when I get back to the office.  (It wasn't
> livelocked; in fact, the server was mostly idle, but responses would
> take seconds rather than milliseconds -- assuming the client could
> even successfully mount the server at all, which the Debian
> automounter frequently refused to do.)

This is very weird and unlikely to come from the 4k mbufs by itself
considering they are in heavy use in the write() path.  Such a high
delay smells like an issue in either the driver dealing with multiple
mbufs per packet or nfs having a problem with it.

> I ended up reverting back to the old kernel (which I managed to lose
> the sources for), and once I get my second server up, I will try to do
> some more testing to see if I can identify the source of the problem.

-- 
Andre