mbuf leakage with nfs/udp (was: mbuf leakage with nfs/zfs)

[Daniel Braniss]

>
> On Feb 28, 2010, at 12:11 PM, Daniel Braniss wrote:
> 
> >> I'm pulling in Robert Watson, who has some familiarity with the UDP
> >> stack/code in FreeBSD.  I'm not sure he'll be a sufficient source of
> >> knowledge for this specific issue since it appears (?) to be specific =to
> >> NFS; Rick Macklem would be a better choice, but as reported, he's =MIA.
> >> 
> >> Robert, are you aware of any changes or implementation issues which
> >> might cause excessive (read: leaking) mbuf use under UDP-based NFS?  =Do
> >> you know of a way folks could determine the source of the leak, =either
> >> via DDB or while the system is live?
> > 
> > I have been runing some tests in a controlled environment.
> > > > server and client are both 64bit Xeon/X5550 @  2.67GHz with 16Gb of =memory
> > FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
> > > > the client is runing latest 8.0 stable
> > the load is created by runing 'make -j32 buildworld' and sleeping 150 =sec.
> > in between runs, this is the straight line you will see in the graphs.
> > Both the src and obj directories are NFS mounted from the server, =regular UFS.
> > > > when server is running 7.2-stable no leakage is seen.
> > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/{tcp,udp}-7.2.ps
> > when server is runing 8.0-stable
> > see ftp://ftp.cs.huji.ac.il/users/danny/freebsd/mbufs/{tcp,udp}-8.0.ps
> > you can see that udp is leaking!
> > > > cheers,
> > 	danny
> > ps: I think the subject should be changed again, removing zfs ...
> > This type of problem (occurs with one client but not another) is almost =always the result of the access pattern of a particular client =triggering a specific (and perhaps single) bug in error-handling. For =example, we might not be properly freeing the received request when =generating an EPERM in an edge case. The hard bit is identifying which =it is. If it's reproducible with UDP, then usually the process is:
> > - Build a minimal test case to trigger the problem -- ideally with as =little complexity as possible.
> - Run netstat -m at the beginning of the test and the end of the test on =the server to count the number of leaked mbufs
> - Run wireshark throughout the test
> - Walk the wireshark trace looking for some error that occurs at about =the same or slightly lower number of times then the number of mbufs =leaked
> - Iterate, narrowing the test case until it's either obvious exactly =what's going on, or you've identified a relatively constrained code path =and can just spot the bug by reading the code
> > It's almost certainly one or a small number of very specific RPCs that =are triggering it -- maybe OpenBSD does an extra lookup, or stat, or =something, on a name that may not exist anymore, or does it sooner than =the other clients. Hard to say, other than to wave hands at the =possibilities.
> > And it may well be we're looking at two bugs: Danny may see one bug, =perhaps triggered by a race condition, but it may be different from the =OpenBSD client-triggered bug (to be clear: it's definitely a FreeBSD =bug, although we might only see it when an OpenBSD client is used =because perhaps OpenBSD also has a bug or feature).
> > Robert=

well, I have further reduced the problem, it happens with NFS/UDP writes.
i'll try the wireshark road, but i'm very rusty with RPC, the other road is to
check the changes, my oldest is from late october (RC2) where it's happening, 
while
Gerrit tried 8-pre from November and worked, so it will be fun
trying to nail it down :-)

cheers,
	danny