kern/51685: Unbounded inode allocation causes kernel to lock up
Michael van Elst
mlelstv at dev.de.cw.net
Fri May 2 05:20:12 PDT 2003
>Synopsis: Unbounded inode allocation causes kernel to lock up
>Arrival-Date: Fri May 02 05:20:09 PDT 2003
>Originator: Michael van Elst
>Release: FreeBSD 4.8-RELEASE i386
System: FreeBSD pg6.dev.de.cw.net 4.8-RELEASE FreeBSD 4.8-RELEASE #0: Thu Apr 3 10:53:38 GMT 2003 root at freebsd-stable.sentex.ca:/usr/obj/usr/src/sys/GENERIC i386
For every cached filesystem object a 'vnode' and an associated
'inode' are allocated in memory. The inodes are allocated in the
'FFS node' memory region.
When a vnode is no longer used it is either explicitely released to a
free list or reclaimed by a kernel thread. The reclaim operation in
vnlru_proc() also releases the associated filesystem data. The release
operation done by calling vput() or vrele() on the other hand leave the
associated filesystem data intact. It is released only when a new vnode
needs to be allocated by recycling a vnode from the free list.
The 'FFS node' region is limited to about 100MB, when it is filled up
the kernel sleeps until some process has freed memory in that region.
When no other process is allocating new vnodes this can only happen
by vnlru_proc() pushing vnodes to the free list. vnlru_proc() however
only runs when the number of vnodes that are not on the free list
exceeds the system parameter 'kern.maxvnodes'. When the inode memory
allocated for the remaining 'kern.maxvnodes' vnodes and for all vnodes
on the free list that are still associated with filesystem data reaches
the 100MB limit the system freezes.
The vfs.vmiodirenable setting only amplifies the problem. Without
directory caching you need to access more files to reach the limit.
FreeBSD5 has the same issue, however the critical memory region
is now handled by UMA and has no artificial limit. It is still limited
to the size of a UMA zone.
- run a FreeBSD4 system with vfs.vmiodirenable=1
- on a FFS filesystem create a directory with about one million files or
- do a 'find -ls' over this directory or wait for some periodic
task that does similar (e.g. weekly/310.locate).
- watch the system slow down and lock up
No fix known, but some ideas:
If vrele() and vput() would simply drop a vnode it could always be
reclaimed by vnlru_proc(). Caching would be done only in the vnode
layer but not also in the ffs layer.
As an alternative vlnru_proc() could use a heuristic that prevents a
resource shortage in the ffs layer, something like 'desiredusedvnodes'
that is checked against the number of vnodes that carry filesystem
More information about the freebsd-bugs