mbuf leakage with nfs/zfs?

[Rick Macklem]

On Sat, 27 Feb 2010, Jeremy Chadwick wrote:

>> I concur.
>> Everything in my network is now on TCP, and there is no mbuf leakage.
>> I just don't get over the 5500 mark, no matter what I throw at it.
>>
>> I do feel that TCP is not as well performing on a local net with Linux,
>> hence the choice for UDP. But TCP is workable as next best.
>
> NFS; Rick Macklem would be a better choice, but as reported, he's MIA.
>

Not exactly MIA, but only able to read email from time to time at this
point. I don't know when I'll be able to do more than that.

So, it does sound like it is UDP specific. Robert mentioned one scenario,
which was an infrequently executed code path that is being tickled and it
has a missing m_freem().

One thing someone could try is switching to the experimental nfs server
("-e" on both mountd and nfsd) and see if the leak goes away. If it does
go away, it is almost certainly the above in the regular nfs server code.

If it doesn't go away, the problem is more likely in the krpc or the
generic udp code. (When I looked at svc_dg.c, I could only spot one
possible leak and you've already determined that patch doesn't help.
The other big difference when using udp on the FreeBSD8 krpc is the
reply cache code. I seem to recall it's an lru cache with a fixed upper
bound, but it might be broken and leaking.

If you change the server to set sp_rcache = NULL in the initialization
function in sys/nfsserver/nfs_srvkrpc.c, I think that disables the replay
cache. You wouldn't want to run this way in production, but it would 
determine if the leak is in it.

Change the 3 lines in nfsrv_init() to:
nfsrv_pool->sp_rcache = NULL;
nfsrv_pool->sp_assign = NULL;
nfsrv_pool->sp_done = NULL;

and I think the krpc replay cache will be disabled.

Good luck with it and please report back if you get to try the above.

I'll get back to committing etc one of these days, rick