Differences in memory handling on systems with/out cache drives

[Erik Stian Tefre]

On 05. nov. 2014 09:53, Karli Sjöberg wrote:
> On Wed, 2014-11-05 at 08:44 +0000, Karli Sjöberg wrote:
>> Hey all!
>>
>> Still investigating the intermittent lockups we are experiencing on our
>> storage systems and have started to compare memory graphs from our
>> Graphite monitoring system. What´s interesting about two of our systems
>> is that they both have the same amount of RAM; 32 GB. But on one of
>> them, I have "zpool remove"'d the cache drives from the pool and have
>> been able to study how different their memory graphs now look like.
>>
>> Also worth noting is that the cache-less system nearly haven´t swapped
>> at all (1112K) since the last stall 20 days ago, while the other system
>> has swapped 78 MB during it´s 48 days of uptime.
>>
>> I´ve attached both screenshots from the two systems, with- and without
>> cache drives, displaying a period of 12 hours.
>>
>> What´s most notable are the characteristic cuts that happen on the
>> cache-less system when ZFS goes in and evicts blocks from ARC that shows
>> as a decrease in "wired" and increase in "free", that just doesn´t
>> happen/looks different in the system with cache drive configured in the
>> pool.
>>
>> What´s your take on this? Are we hitting bug: 187594 perhaps? How can we
>> know?
Take a look at the recent thread on freebsd-stable@, subject "ARC size 
limit". Possibly related issues have been patched in stable/10, the 
mailing list thread includes a patch for releng/10.1.

--
Erik