vnode leak in FFS code ... ?
Marc G. Fournier
scrappy at hub.org
Sat Sep 4 09:56:23 PDT 2004
On Fri, 3 Sep 2004, Julian Elischer wrote:
> Marc G. Fournier wrote:
>> Just as a followup to this ... the server crashed on Thursday night around
>> 22:00ADT, only just came back up after a very long fsck ... with all 62 VMs
>> started up, and 1008 processes running, vnodes currently look like:
> are you using nullfs at all on your vms?
No, I stop'd using that over a year ago, figuring that it was exasperating
the problems we were having back then ... the only thing we did use nullfs
at that time was so that we could 'identify' which files were specific to
a VM, vs a file on the template ... we moved to using nfs to do the same
The only thing we use is unionfs, and nfs ...
Basically, we do a 'mount_union -b <template> <vm>', where template is a
shared file system containing common applications, in order to reduce
overall disk space being used by each client. So, for instance, on one of
our servers we have a 'template' VM that, when we need to add/upgrade an
application, we start up, log into and install from ports ... then, we
rsync that template to the live server(s) so that those apps are available
within all VMs ...
We then use NFS to mount the 'base' file system for each VM, that contains
only the changed files that are specific to the VM (ie. config files, any
apps the client happens to have installed, etc) and use that to determine
storage usage ...
there is only one NFS mount point, that covers the whole file system, we
dont do a mount per VM or anything like that ...
So, in the case of the system that has risen quite high on the vnode
count, with 60 VMs ... there would be:
5 UFS mounts
- /, /var, /tmp, /usr and /vm
- /vm is where the virtual machines run off of
1 NFS mount
60 UNIONFS mount points
- one for each VM
60 procfs mount points
- one for each VM
Thanks to the work that David and Tor put in last summer on vnlru, this
works quite well, with the occasional crash when a 'fringe bug' gets
tweaked ... our record uptime on a server, in this configuration, so far
is 106days ...
The part that hurts the most is that the longer the server is up and
running, the greater the chance of having a 12+hr fsck run due to all the
ZERO LENGTH DIRECTORYs :(
Whenever I get a good core dump, I try and post a report to GNaTs, but
between everyone focusing on 5.x, and those crying "unionfs is broken",
they tend to sit in limbo ... altho most of the bugs I am able to find
most likely exist in 5.x code as well, and fixing them would go one more
step towards improving unionfs ...
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy at hub.org Yahoo!: yscrappy ICQ: 7615664
More information about the freebsd-current