Limits on jumbo mbuf cluster allocation

[Rick Macklem]

Andre Oppermann wrote:
> On 10.03.2013 07:04, Garrett Wollman wrote:
> > <<On Fri, 8 Mar 2013 12:13:28 -0800, Jack Vogel <jfvogel at gmail.com>
> > said:
> >
> >> Yes, in the past the code was in this form, it should work fine
> >> Garrett,
> >> just make sure
> >> the 4K pool is large enough.
> >
> > [Andre Oppermann's patch:]
> >>> if (adapter->max_frame_size <= 2048)
> > adapter-> rx_mbuf_sz = MCLBYTES;
> >>> - else if (adapter->max_frame_size <= 4096)
> >>> + else
> > adapter-> rx_mbuf_sz = MJUMPAGESIZE;
> >>> - else if (adapter->max_frame_size <= 9216)
> >>> - adapter->rx_mbuf_sz = MJUM9BYTES;
> >>> - else
> >>> - adapter->rx_mbuf_sz = MJUM16BYTES;
> >
> > So I tried exactly this, and it certainly worked insofar as only 4k
> > clusters were allocated, but NFS performance went down precipitously
> > (to fewer than 100 ops/s where normally it would be doing 2000
> > ops/s). I took a tcpdump while it was in this state, which I will
> > try
> > to make some sense of when I get back to the office. (It wasn't
> > livelocked; in fact, the server was mostly idle, but responses would
> > take seconds rather than milliseconds -- assuming the client could
> > even successfully mount the server at all, which the Debian
> > automounter frequently refused to do.)
> 
> This is very weird and unlikely to come from the 4k mbufs by itself
> considering they are in heavy use in the write() path. Such a high
> delay smells like an issue in either the driver dealing with multiple
> mbufs per packet or nfs having a problem with it.
> 
I am not aware of anything within the NFS server that would care. The
code simply believes the m_len field.

  --> However, this is a good way to reduce server load. At 100ops/sec
      I'd think you shouldn't have any server resource exhaustion issues.
      --> Problem solved ;-);-)

rick

> > I ended up reverting back to the old kernel (which I managed to lose
> > the sources for), and once I get my second server up, I will try to
> > do
> > some more testing to see if I can identify the source of the
> > problem.
> 
> --
> Andre
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"