Where have all the vnodes gone?

[Linda Messerschmidt]

(Reposted from freebsd-questions due to no replies.)

With the last few releases, I've noticed a distinct trend toward
disappearing vnodes on one of the machines I look after.

This machine isn't doing a whole lot.  It runs a couple of small web
sites, and once an hour it rsync's some files from one NFS mount to
another, but the rsync doesn't stay running; it restarts every hour
and runs for 10-15 minutes.

I set it to log the number of vnodes every ten minutes and this is what I got:

00:47:59 vfs.numvnodes: 39337
00:57:59 vfs.numvnodes: 40568
01:07:59 vfs.numvnodes: 44554
01:17:59 vfs.numvnodes: 52141
01:27:59 vfs.numvnodes: 55713
01:37:59 vfs.numvnodes: 58643
01:47:59 vfs.numvnodes: 60792
01:57:59 vfs.numvnodes: 67130
02:07:59 vfs.numvnodes: 76035
02:17:59 vfs.numvnodes: 84349
02:27:59 vfs.numvnodes: 92187
02:37:59 vfs.numvnodes: 98114
02:47:59 vfs.numvnodes: 116854
02:57:59 vfs.numvnodes: 124842
03:07:59 vfs.numvnodes: 164173
03:17:59 vfs.numvnodes: 172257
03:27:59 vfs.numvnodes: 178388
03:37:59 vfs.numvnodes: 183066
03:47:59 vfs.numvnodes: 190092
03:57:59 vfs.numvnodes: 198322
04:07:59 vfs.numvnodes: 204598
04:17:59 vfs.numvnodes: 208311
04:27:59 vfs.numvnodes: 214207
04:37:59 vfs.numvnodes: 221028
04:47:59 vfs.numvnodes: 227792
04:57:59 vfs.numvnodes: 233214
05:07:59 vfs.numvnodes: 240112
05:17:59 vfs.numvnodes: 247572
05:27:59 vfs.numvnodes: 256090
05:37:59 vfs.numvnodes: 262720
05:47:59 vfs.numvnodes: 269755
05:57:59 vfs.numvnodes: 274986
06:07:59 vfs.numvnodes: 279879
06:17:59 vfs.numvnodes: 287039
06:27:59 vfs.numvnodes: 291984
06:37:59 vfs.numvnodes: 294267
06:47:59 vfs.numvnodes: 296658
06:57:59 vfs.numvnodes: 299086
07:07:59 vfs.numvnodes: 301825
07:17:59 vfs.numvnodes: 309060
07:27:59 vfs.numvnodes: 312955
07:37:59 vfs.numvnodes: 317400
07:47:59 vfs.numvnodes: 320047

At that point the machine crashed with:

panic: kmem_malloc(16384): kmem_map too small: 334745600 total allocated

If I don't tune kern.maxvnodes up to the point where it panics, then
eventually it runs out of vnodes and all sorts of stuff gets stuck in
vlruwk.

The machine in question is running 7.2-RELEASE-p3, but I already
upgraded it from 7.1 trying to get this to go away, so it's a problem
that's been around for awhile.

My guess is that they're leaking in the kernel somewhere because of
the rsync, because there's just not much else going on, but unless I
can figure out how many vnodes are being used on a per-process basis,
I can't make any headway on proving or disproving that.  I do know
that according to fstat, there are only 1000-1500 descriptors open at
any given time, and kern.openfiles ranges 250-500.  So I'm just
mystified what the other 300000+ could be.

Is there any way to figure out where all these vnodes are going?

TIA for any advice!