vnodes - is there a leak? where are they going?

[David Schultz]

On Tue, Aug 31, 2004, Marc G. Fournier wrote:
> 
> I have two servers, both running 4.10 of within a few days (Aug 5 for 
> venus, Aug 7 for neptune) ... both running jail environments ... one with 
> ~60 running, the other with ~80 ... the one with 60 has been running for 
> ~25 days now, and is at the border of running out of vnodes:
> 
> Aug 31 20:58:00 venus root: debug.numvnodes: 519920 - debug.freevnodes: 
> 11058 - debug.vnlru_nowhere: 256463 - vlrup
> Aug 31 20:59:01 venus root: debug.numvnodes: 519920 - debug.freevnodes: 
> 13155 - debug.vnlru_nowhere: 256482 - vlrup
> Aug 31 21:00:03 venus root: debug.numvnodes: 519920 - debug.freevnodes: 
> 13092 - debug.vnlru_nowhere: 256482 - vlruwt
> 
> while the other one has been up for ~1 days, but is using alot less, for 
> more processes:
> 
> Aug 31 20:58:00 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 
> 208655 - debug.vnlru_nowhere: 0 - vlruwt
> Aug 31 20:59:00 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 
> 208602 - debug.vnlru_nowhere: 0 - vlruwt
> Aug 31 21:00:03 neptune root: debug.numvnodes: 344062 - debug.freevnodes: 
> 208319 - debug.vnlru_nowhere: 0 - vlruwt
> 
> I've tried shutting down all of the VMs on venus, and umount'd all of the 
> unionfs mounts, as well as the one nfs mount we have ... the above #s are 
> after the VMs (and mounts are recreated ...
> 
> Now, my understanding of the vnodes is that for every file opened, a vnode 
> is created ... in my case, since I'm using unionfs, there are two vnodes 
> per file ... if it possible that there are 'stale' vnodes that aren't 
> being freed up?  Is there some way of 'viewing' the vnode structure?
> 
> For instance, fstat shows:
> 
> venus# fstat | wc -l
>    19531
> 
> So, obviously it isn't just open files that I'm dealing with here, for 
> even if I double that, that is nowhere near 519920 ...
> 
> So, where else are the vnodes going?  Is there a 'leak'?  What can I look 
> at to try and narrow this down / provide more information?

First of all, use 'fstat -m' to ensure that you're counting all
open files.  Second, I believe it is normal for the number of
files that fstat reports to be lower than the number of vnodes
actually allocated, since unreferenced vnodes (which don't show up
in fstat) are cached.

It's a bit worrisome that debug.vnlru_nowhere is large on one
machine and 0 on the other.  That number says that the system
tried to reclaim some unreferenced vnodes, but didn't find any.  I
don't know whether this indicates a leak, or merely a
vnode-intensive process.

> Even some way of determining a specific process that is sucking back alot 
> of them, to move that to a different machine ... ?

'fstat -mu' or 'fstat -mp' might be helpful in tracking down which
user or process is eating up your vnodes.