Disk full / NFS, df, and du

[Dan Nelson]

In the last episode (May 17), Eric Anderson said:
> Dan Nelson wrote:
> >In the last episode (May 16), Eric Anderson said:
> >>I have a few large NFS file servers, holding about 1Tb of diskspace
> >>each.  I break those logical disks (it's on a hardware RAID) into
> >>partitions, and share them.  My users fill up the partitions often
> >>enough, and when they do, they rm entire directory trees to free
> >>the space.  They use du to determine how much space is in a
> >>directory and how much they are hogging.
> >>
> >>The problem I'm having is, after they do the rm's, it doesn't free
> >>the disk space.  df shows it still being used, but du claims their
> >>directories are empty.
> >>
> >>If I reboot the file server, the space magically appears.
> >
> >Does a du on server itself show files?  How about "lsof +L1"?  The
> >NFS protocol doesn't allow clients to unlink files they have open,
> >so FreeBSD clients (at least) rename open files that are unlinked to
> >.nfs##### until the last process closes the file, and then they delete
> >it.  If you've got unlinked files held open, it's got to be on the
> >server itself.
> 
> lsof +L1 shows nothing.. any more ideas?

Actually now that I think about it I have seen similar symptoms on my
4.8 servers where a volume would fill up but no amount of deleting
would help until a reboot.  It happens very infrequently (maybe twice a
year), and I assume it's due to a filesystem-full bug in the
softupdates code since on 4.x free space isn't really free until
softupdates has flushed its updates to disk, and that could take a
while.  Try dropping the kern.{file,dir,meta}delay sysctls down to like
{7,6,5} seconds and see if that does anything for you.

-- 
	Dan Nelson
	dnelson at allantgroup.com