Limits on jumbo mbuf cluster allocation

[Rick Macklem]

Garrett Wollman wrote:
> <<On Fri, 08 Mar 2013 08:54:14 +0100, Andre Oppermann
> <andre at freebsd.org> said:
> 
> > [stuff I wrote deleted]
> > You have an amd64 kernel running HEAD or 9.x?
> 
> Yes, these are 9.1 with some patches to reduce mutex contention on the
> NFS server's replay "cache".
> 
The cached replies are copies of the mbuf list done via m_copym().
As such, the clusters in these replies won't be free'd (ref cnt -> 0)
until the cache is trimmed (nfsrv_trimcache() gets called after the
TCP layer has received an ACK for receipt of the reply from the client).

If reducing the size to 4K doesn't fix the problem, you might want to
consider shrinking the tunable vfs.nfsd.tcphighwater and suffering
the increased CPU overhead (and some increased mutex contention) of
calling nfsrv_trimcache() more frequently.
(I'm assuming that you are using drc2.patch + drc3.patch. If you are
 using one of ivoras@'s variants of the patch, I'm not sure if the
 tunable is called the same thing, although it should have basically
 the same effect.)

Good luck with it and thanks for running on the "bleeding edge" so
these issues get identified, rick

> > Jumbo pages come directly from the kernel_map which on amd64 is
> > 512GB.
> > So KVA shouldn't be a problem. Your problem indeed appears to come
> > physical memory fragmentation in pmap.
> 
> I hadn't realized that they were physically contiguous, but that makes
> perfect sense.
> 
> > pages. Also since you're doing NFS serving almost all memory will be
> > in use for file caching.
> 
> I actually had the ZFS ARC tuned down to 64 GB (out of 96 GB physmem)
> when I experienced this, but there are plenty of data structures in
> the kernel that aren't subject to this limit and I could easily
> imagine them checkerboarding physical memory to the point where no
> contiguous three-page allocations were possible.
> 
> -GAWollman
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"