Memory leak in ZFS?

[Martin Matuska]

I am using ZFS and sendfile(2) with apache, lighttpd, nginx and on
FreeBSD 8.2-RC (v15 or with ZFS v28 patch). The systems concerned are
quite busy.
E.g. lighttpd servers delivering 30-50 Megabytes per second (half a
gigabit) or samba servers delivering to hundreds of users.

Since FreeBSD 8.1, really a lot of bugfixes have been ported to ZFS
including very important sendfile(2) fixes and I highly recommend
upgrading to 8.2 or 8-STABLE.
As of v28, I am using the patch for 8-STABLE on several production servers.

When 8.2-RELEASE is out, I plan to put a v28 patch + v28 binary upgrade
kit on my webpage.

As of the memory usage shown by top(1), I have noticed that memory used
by some programs using IPC shared memory (e.g. postgresql) do not show
up in the counts.
After stopping the program the count is "fixed" again.

Dňa 08.02.2011 11:27, Jeremy Chadwick wrote / napísal(a):
> On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote:
>> W dniu 2011-02-07 22:37, Emil Muratov pisze:
>>>> For the past few weeks, I noticed that the amount of memory
>>>> reported in top
>>>> (sum of active, inact, wired, cache buf and free) keeps
>>>> decreasing as the
>>>> uptime increases. I can't pinpoint to when I first noticed this,
>>>> as I have
>>>> updated the system a few times just in case this has been fixed.
>>> Yes, I have the same issue on my home file storage. My system is
>>> 8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives.
>>> After updating to stable a couple of days ago I noticed that the
>>> system leaks memory very fast. Checking here and there I found
>>> that the issue concerns sendfile (yep, again!).
>>>
>>> How to reproduce:
>>> Configure samba with aio and sendfile (mine is version 3.5.6)
>>>
>>> smb.conf
>>> [global]
>>> use sendfile=true
>>> aio read size = 16384
>>>
>>> Download a couple of large samba shared files (8-10 gigs).
>>>
>>>
>>> While downloading files I can see that memory decreazes to nowhere
>>> very-very fast, several MBs per second! First it drains free mem,
>>> than active and inactive, than comes wired until the whole system
>>> commits suicide suffocating itself to the death.
>>> The only way to free memory is to reboot the system. I can't
>>> unload zfs module like PJD suggested to do, 'cause my root is on
>>> zfs :(
>>> I'll try to make a bootable flash and move root to the flash to
>>> try to unload module and what will happen.
>>>
>>> Everything was OK in stable before the new year, sendfile used to
>>> pump free and wired memory to inactive than slowly reclaiming it
>>> back. But it seems something was changed after NY holydays?
>> I'm glad someone else finally picked that problem, so there's
>> appareantly no memory-eating ghost in my machine ;)
>> Here's my thread on stable list about this issue:
>> http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html
>>
>> And in fact, PC reported in thread above is also SAMBA server with
>> aio/sendfile enabled and ZFS.
>>
>> I would be happy testing some patches if necessary, because until
>> now I need to monitor memory and reboot this server before it dies.
> The source and build date of your kernel will matter greatly here.
>
> I can't speak about the memory utilisation aspect, but I tend to disable
> sendfile everywhere possible when ZFS is in use on a system.  The reason
> is based on something I and another user experienced back in October
> 2010 pertaining to sendfile() on ZFS locking up processes (making them
> unkillable).  See here[1] for details; this problem has since been
> fixed[2] (look for commits around October).  You'll also find some
> commits that went through in November pertaining to ZFS and sendfile.
> This is why I said the date of your kernel/sources matters.  :-)
>
> The issue I referenced in [1] is not related to memory utilisation, but
> does indicate use of sendfile with ZFS may be a bad idea (by this I
> mean, there may be aspects of its implementation when mixed with ZFS
> that have been overlooked).
>
> Simple test: if you disable use of sendfile (but not AIO) in Samba, does
> the problem go away?
>
> Comparatively, Apache out-of-the-box defaults to mmap() and sendfile()
> being disabled.  BSD ftpd, however, is known to use sendfile
> exclusively.
>
> [1]: http://lists.freebsd.org/pipermail/freebsd-fs/2010-October/thread.html#9729
> [2]: http://www.freshbsd.org/?branch=RELENG_8&project=freebsd&committer=&module=src&q=sendfile
>