Use of contiguous physical memory in cxgbe driver

[Garrett Wollman]

<<On Wed, 12 Feb 2014 14:46:19 -0500, John Baldwin <jhb at freebsd.org> said:

> Is this because UMA keeps lots of mbufs cached in your workload?
> The physmem buddy allocator certainly seeks to minimize
> fragmentation.  However, it can't go yank memory out of UMA caches
> to do so.

It's not just UMA caches: there are TCP queues, interface queues, the
NFS request "cache", and elsewhere.  I first discovered this problem
in the NFS context: what happens is that you build up very large TCP
send buffers (NFS forces the socket buffers to 2M) for many clients
(easy if the server is dedicated 10G and the clients are all on shared
1G links).  The NIC is eventually unable to replenish its receive
ring, and everything just stops.  Eventually, the TCP connections time
out, the buffers are freed, and the server mysteriously starts working
again.  (Actually, the last bit never happens in production.  It's
more like: Eventually, the users start filing trouble tickets, then
Nagios starts paging the sysadmins, then someone does a hard reset
because that's the fastest way to recover.  And then they blame me.)

-GAWollman