Memory leak in ZFS?

[Jeremy Chadwick]

On Tue, Feb 08, 2011 at 10:24:11AM +0100, Bartosz Stec wrote:
> W dniu 2011-02-07 22:37, Emil Muratov pisze:
> >
> >>For the past few weeks, I noticed that the amount of memory
> >>reported in top
> >>(sum of active, inact, wired, cache buf and free) keeps
> >>decreasing as the
> >>uptime increases. I can't pinpoint to when I first noticed this,
> >>as I have
> >>updated the system a few times just in case this has been fixed.
> >
> >Yes, I have the same issue on my home file storage. My system is
> >8.1 amd64, 2G ram, zfs on root raidz with 4x1,5T drives.
> >After updating to stable a couple of days ago I noticed that the
> >system leaks memory very fast. Checking here and there I found
> >that the issue concerns sendfile (yep, again!).
> >
> >How to reproduce:
> >Configure samba with aio and sendfile (mine is version 3.5.6)
> >
> >smb.conf
> >[global]
> >use sendfile=true
> >aio read size = 16384
> >
> >Download a couple of large samba shared files (8-10 gigs).
> >
> >
> >While downloading files I can see that memory decreazes to nowhere
> >very-very fast, several MBs per second! First it drains free mem,
> >than active and inactive, than comes wired until the whole system
> >commits suicide suffocating itself to the death.
> >The only way to free memory is to reboot the system. I can't
> >unload zfs module like PJD suggested to do, 'cause my root is on
> >zfs :(
> >I'll try to make a bootable flash and move root to the flash to
> >try to unload module and what will happen.
> >
> >Everything was OK in stable before the new year, sendfile used to
> >pump free and wired memory to inactive than slowly reclaiming it
> >back. But it seems something was changed after NY holydays?
> 
> I'm glad someone else finally picked that problem, so there's
> appareantly no memory-eating ghost in my machine ;)
> Here's my thread on stable list about this issue:
> http://lists.freebsd.org/pipermail/freebsd-stable/2011-January/061247.html
> 
> And in fact, PC reported in thread above is also SAMBA server with
> aio/sendfile enabled and ZFS.
> 
> I would be happy testing some patches if necessary, because until
> now I need to monitor memory and reboot this server before it dies.

The source and build date of your kernel will matter greatly here.

I can't speak about the memory utilisation aspect, but I tend to disable
sendfile everywhere possible when ZFS is in use on a system.  The reason
is based on something I and another user experienced back in October
2010 pertaining to sendfile() on ZFS locking up processes (making them
unkillable).  See here[1] for details; this problem has since been
fixed[2] (look for commits around October).  You'll also find some
commits that went through in November pertaining to ZFS and sendfile.
This is why I said the date of your kernel/sources matters.  :-)

The issue I referenced in [1] is not related to memory utilisation, but
does indicate use of sendfile with ZFS may be a bad idea (by this I
mean, there may be aspects of its implementation when mixed with ZFS
that have been overlooked).

Simple test: if you disable use of sendfile (but not AIO) in Samba, does
the problem go away?

Comparatively, Apache out-of-the-box defaults to mmap() and sendfile()
being disabled.  BSD ftpd, however, is known to use sendfile
exclusively.

[1]: http://lists.freebsd.org/pipermail/freebsd-fs/2010-October/thread.html#9729
[2]: http://www.freshbsd.org/?branch=RELENG_8&project=freebsd&committer=&module=src&q=sendfile

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |