kern/187594: [zfs] [patch] ZFS ARC behavior problem and fix

[Andriy Gapon]

The following reply was made to PR kern/187594; it has been noted by GNATS.

From: Andriy Gapon <avg at FreeBSD.org>
To: bug-followup at FreeBSD.org, karl at fs.denninger.net
Cc:  
Subject: Re: kern/187594: [zfs] [patch] ZFS ARC behavior problem and fix
Date: Tue, 18 Mar 2014 17:15:05 +0200

 Karl Denninger <karl at fs.denninger.net> wrote:
 > ZFS can be convinced to engage in pathological behavior due to a bad
 > low-memory test in arc.c
 > 
 > The offending file is at
 > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c; it allegedly
 > checks for 25% free memory, and if it is less asks for the cache to shrink.
 > 
 > (snippet from arc.c around line 2494 of arc.c in 10-STABLE; path
 > /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs)
 > 
 > #else /* !sun */
 > if (kmem_used() > (kmem_size() * 3) / 4)
 > return (1);
 > #endif /* sun */
 > 
 > Unfortunately these two functions do not return what the authors thought
 > they did. It's clear what they're trying to do from the Solaris-specific
 > code up above this test.
 
 No, these functions do return what the authors think they do.
 The check is for KVA usage (kernel virtual address space), not for physical memory.
 
 > The result is that the cache only shrinks when vm_paging_needed() tests
 > true, but by that time the system is in serious memory trouble and by
 
 No, it is not.
 The description and numbers here are a little bit outdated but they should give
 an idea of how paging works in general:
 https://wiki.freebsd.org/AvgPageoutAlgorithm
 
 > triggering only there it actually drives the system further into paging,
 
 How does ARC eviction drives the system further into paging?
 
 > because the pager will not recall pages from the swap until they are next
 > executed. This leads the ARC to try to fill in all the available RAM even
 > though pages have been pushed off onto swap. Not good.
 
 Unused physical memory is a waste.  It is true that ARC tries to use as much of
 memory as it is allowed.  The same applies to the page cache (Active, Inactive).
 Memory management is a dynamic system and there are a few competing agents.
 
 It is hard to correctly tune that system using a large hummer such as your
 patch.  I believe that with your patch ARC will get shrunk to its minimum size
 in due time.  Active + Inactive will grow to use the memory that you are denying
 to ARC driving Free below a threshold, which will reduce ARC.  Repeated enough
 times this will drive ARC to its minimum.
 
 Also, there are a few technical problems with the patch:
 - you don't need to use sysctl interface in kernel, the values you need are
 available directly, just take a look at e.g. implementation of vm_paging_needed()
 - similarly, querying vfs.zfs.arc_freepage_percent_target value via
 kernel_sysctlbyname is just bogus; you can use percent_target directly
 - you don't need to sum various page counters to get a total count, there is
 v_page_count
 
 Lastly, can you try to test reverting your patch and instead setting
 vm.lowmem_period=0 ?
 
 -- 
 Andriy Gapon