held file reference issue with ZFS and nullfs

[Florian Schulze]

Hi!

On 6 Sep 2019, at 14:43, Konstantin Belousov wrote:

Thanks for the reply ...

> On Fri, Sep 06, 2019 at 12:44:46PM +0200, Florian Schulze wrote:
>> Hi!
>>
>> Since FreeBSD 12 (updated from 10.3, I skipped 11.x completely, the 
>> box
>> started around 9.3) I have the issue that ZFS is not freeing up space
>> for some deleted files. The filesystems where this happens are 
>> mounted
>> into multiple jails via nullfs. Only one jail has write access, the
>> others are read only. When files are deleted the space for them is 
>> not
>> freed. I can still see their objects via zdb. When I unmount one of 
>> the
>> read only nullfs mounts the space is freed and the objects released.
>>
>> I already used lsof, procstat and fstat to see if any process still 
>> has
>> a reference to the file, but that is not the case. But it seems to
>> matter which nullfs mount is unmounted, it is always one of the read
>> only ones. The processes which access the read only mounts are
>> completely different, it only seems to matter that the files are 
>> opened
>> at all. Killing the processes doesn't help, only unmounting the 
>> nullfs.
> There were some bugs in past where nullfs referenced a lower vnode but
> did not dereferenced it.

Could you be more concrete about "past"? In the past of the 12.0 
version, or during 11.x?

>> I have one zfs filesystem where this happens quite often, one were it
>> happens sometimes and a few others which have a similar setup and 
>> where
>> I never noticed it (though the average file size on them is smaller).
>>
>> I asked in #freebsd about this and koobs said I should write to this
>> list and CC kib at freebsd.org and mgj at freebsd.org
>> He also did a quick look at the nullfs changes between 10.3 and 12.0 
>> and
>> spotted the following change, which he said I should mention as well:
>> https://github.com/freebsd/freebsd/commit/82f9c275c43da09f404546cceeff187a90ecc573#diff-81e7d6520611101890dd6425324dd8f8
>>
>> Is there a known bug there? Could the stale NFS handle cause the 
>> leak?
>> Where is that NFS handle coming from?
>
> So what is the exact version of your system ?  If 12.0, upgrade kernel
> to latest stable/12 and see if it helps with the leak.

I'm on 12.0-RELEASE-p3. I will probably be able to try 12.0-RELEASE-p10 
during the weekend. Not sure I will find the time to try and compile my 
own kernel, or is there a way to do it using freebsd-update by using a 
snapshot? If not, I'd probably rather wait for 12.1.

Regards,
Florian Schulze