memory exhaustion on 10.1 AMD64 ZFS storage system

[Mark Martinec]

Joseph Mingrone wrote:
> We've had this storage system running 9.x without problems.  After
> upgrading to 10.1 we've seen "out of swap space" messages in the logs.
> Dec 13 04:29:12 storage2 kernel: pid 723 (rpc.statd), uid 0, was 
> killed:
> out of swap space [...]
> What's the best way to determine if this is a ZFS problem?  I've read 
> in
> the 10.1 release notes that vfs.zfs.zio.use_uma has been re-enabled.
> Has this caused anyone problems with 10.1?

Aggressiveness of ARC in 10.0 hurt us pretty badly when switching from
9.2 to 10.0.  The ARC / UMA greediness for memory was causing excessive
swapping out of still active processes, while keeping ARC luxuriously
bathing in memory. The situation in 10.1 may have been slightly 
improved,
although it seems the fix still has not been committed:

Bug 187594 - [zfs] [patch] ZFS ARC behavior problem and fix
   https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594

I'd really like to see a solution committed in time for 10.2.


Mark Schouten wrote:
> Do you have compression enabled and L2ARC? There's this bug that leaks
> memory via L2ARC, causing all memory to run out, pushing out the ARC
> and (probably) causes crashes and lots of swap usage..
> 
> I think this is about that bug:
> https://github.com/freebsd/freebsd/commit/b98f85d480b770e34d5e08c66dbc668bd5548bdc

That too (unrelated to ARC / UMA greediness for memory).

   Mark