held file reference issue with ZFS and nullfs
Florian Schulze
mail at florian-schulze.net
Fri Sep 13 15:18:09 UTC 2019
On 6 Sep 2019, at 15:18, Florian Schulze wrote:
> Hi!
>
> On 6 Sep 2019, at 14:43, Konstantin Belousov wrote:
>
> Thanks for the reply ...
>
>> On Fri, Sep 06, 2019 at 12:44:46PM +0200, Florian Schulze wrote:
>>> Hi!
>>>
>>> Since FreeBSD 12 (updated from 10.3, I skipped 11.x completely, the
>>> box
>>> started around 9.3) I have the issue that ZFS is not freeing up
>>> space
>>> for some deleted files. The filesystems where this happens are
>>> mounted
>>> into multiple jails via nullfs. Only one jail has write access, the
>>> others are read only. When files are deleted the space for them is
>>> not
>>> freed. I can still see their objects via zdb. When I unmount one of
>>> the
>>> read only nullfs mounts the space is freed and the objects released.
>>>
>>> I already used lsof, procstat and fstat to see if any process still
>>> has
>>> a reference to the file, but that is not the case. But it seems to
>>> matter which nullfs mount is unmounted, it is always one of the read
>>> only ones. The processes which access the read only mounts are
>>> completely different, it only seems to matter that the files are
>>> opened
>>> at all. Killing the processes doesn't help, only unmounting the
>>> nullfs.
>> There were some bugs in past where nullfs referenced a lower vnode
>> but
>> did not dereferenced it.
>
> Could you be more concrete about "past"? In the past of the 12.0
> version, or during 11.x?
>
>>> I have one zfs filesystem where this happens quite often, one were
>>> it
>>> happens sometimes and a few others which have a similar setup and
>>> where
>>> I never noticed it (though the average file size on them is
>>> smaller).
>>>
>>> I asked in #freebsd about this and koobs said I should write to this
>>> list and CC kib at freebsd.org and mgj at freebsd.org
>>> He also did a quick look at the nullfs changes between 10.3 and 12.0
>>> and
>>> spotted the following change, which he said I should mention as
>>> well:
>>> https://github.com/freebsd/freebsd/commit/82f9c275c43da09f404546cceeff187a90ecc573#diff-81e7d6520611101890dd6425324dd8f8
>>>
>>> Is there a known bug there? Could the stale NFS handle cause the
>>> leak?
>>> Where is that NFS handle coming from?
>>
>> So what is the exact version of your system ? If 12.0, upgrade
>> kernel
>> to latest stable/12 and see if it helps with the leak.
>
> I'm on 12.0-RELEASE-p3. I will probably be able to try
> 12.0-RELEASE-p10 during the weekend. Not sure I will find the time to
> try and compile my own kernel, or is there a way to do it using
> freebsd-update by using a snapshot? If not, I'd probably rather wait
> for 12.1.
It still happens with 12.0-RELEASE-p10. Unless there is a pre-built
kernel to test, I'll wait for 12.1.
Regards,
Florian Schulze
More information about the freebsd-fs
mailing list