Limits on jumbo mbuf cluster allocation

[Rick Macklem]

Andre Oppermann wrote:
> On 11.03.2013 17:05, Garrett Wollman wrote:
> > In article <513DB550.5010004 at freebsd.org>, andre at freebsd.org writes:
> >
> >> Garrett's problem is receive side specific and NFS can't do much
> >> about it.
> >> Unless, of course, NFS is holding on to received mbufs for a longer
> >> time.
> >
> > Well, I have two problems: one is running out of mbufs (caused, we
> > think, by ixgbe requiring 9k clusters when it doesn't actually need
> > them), and one is livelock. Allowing potentially hundreds of clients
> > to queue 2 MB of requests before TCP pushes back on them helps to
> > sustain the livelock once it gets started, and of course those
> > packets
> > will be of the 9k jumbo variety, which makes the first problem worse
> > as well.
> 
> I think that TCP, or rather the send socket buffer, currently doesn't
> push back at all but simply accepts everything that gets thrown at it.
> This obviously is a problem and the NFS server seems to depend
> somewhat
> on that by requiring atomicity on a RPC send. I have to trace the mbuf
> path through NFS to the socket to be sure. The code is slightly opaque
> though.
> 
Yes, I think you are correct that when NFS sends RPC messages over TCP,
they just get queued via sbappendstream(). For some reason I thought the
krpc used sosend_generic(), whereas I just looked and it just uses sosend(),
which does nothing except call tcp_user_send() { for TCP sockets, of course }.

I tend to agree with Garrett that this is ok for the NFS server, since once
it has done the work of generating a reply, why should the nfsd thread get
stuck trying to queue the reply for the client. For the NFS client, it isn't
quite so obvious, but after queuing a request to be sent, it will sit waiting
for the reply, so it will "see" the NFS server's slow response. (Anyhow,
Garrett couldn't care less about the FreeBSD NFS client, since he isn't using
it.;-)

rick

> --
> Andre
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"