Fwd: Strange ZFS problem, filesystem claims to be full when clearly not full

[Torbjorn Kristoffersen]

On Thu, Sep 30, 2010 at 5:47 PM, Andriy Gapon <avg at icyb.net.ua> wrote:
> on 30/09/2010 17:48 Jeremy Chadwick said the following:
>> On Thu, Sep 30, 2010 at 04:38:46PM +0200, Oliver Fromme wrote:
>>> Danny Carroll <fbsd at dannysplace.net> wrote:
>>>  > [...]
>>>  > It certainly smells like a process still writing to a file that is unlinked.
>>>  > I wonder if it would show up with lsof.
>>>
>>> If it's a file that was unlinked that is still held open by
>>> a process, then lsof will definitely list it.  The command
>>>
>>> # lsof +L1
>>>
>>> lists all open files with a link count of zero.  You can
>>> restrict it to a certain file system like this:
>>>
>>> # lsof +aL1 /var
>>>
>>> Of course, lsof won't list the file name because the file
>>> doesn't have a name anymore.  But it lists the process by
>>> name, PID and user, the file system and the file size.
>>
>> Can someone explain how use of lsof in this regard is different than use
>> of fstat(1) like I originally mentioned?  Does lsof do something more
>> thorough or differently that what fstat does?
>
> I believe that there is no reason to prefer lsof except for those who spent more
> time with Linux than with FreeBSD.
>

I tried fstat earlier and now I tried lsof as suggested.  Doing lsof
+L1 only gave me:
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NLINK   NODE NAME
mysqld  1030 mysql    4u  VREG   0,99        0     0 800965 / (/dev/mirror/root)
mysqld  1030 mysql    5u  VREG   0,99        0     0 800969 / (/dev/mirror/root)
mysqld  1030 mysql    6u  VREG   0,99        0     0 800970 / (/dev/mirror/root)
....

Basically, it only gives me mysqld which runs outside the jails.
Nothing else was listed.

I noticed that the filesystem has stopped growing now though, so that
may also be the reason
why lsof does not show anything anymore. The "du -sh /jails/rb.org"
still gives a low usage value.
Also, this is the output from df -h (I've since resized the ZFS quota
to make the filesystem bigger for this jail):

tpool/rb.org      200G    111G     89G    56%    /jails/rb.org

If the process causing this is gone, or is working correctly (seeing
that the fs is no longer growing, I hope),
can dead unlinked files still remain, is there a way to purge them?