ARC size constantly shrinks, then ZFS slows down extremely

[K. Macy]

>>
>> Currently, the inactive page queue will grow until ARC is shrunk to  
>> arc_min.
>>
>> I think I'll probably spend some time making the ARC play better  
>> with the page cache this week. Unfortunately, under heavy memory  
>> pressure when competing with UFS the ARC will degrade to LRU, but I  
>> think that is still an improvement over the current static sizing  
>> with low and high water marks.
> Will setting ARC's minimum size help here? (I will try)
>
> For me, it's OK, and I think it's generally not a problem, if  
> somebody uses UFS as well.
> Is it possible to merge the memory management of the two, or they  
> are completely different beasts?

To some degree it is possible to merge them by partly backing the arc  
from the page cache. This would allow for a fair amount of auto- 
tuning. However, it isn't possibly to completely merge them - the ARC  
is a virtual device block cache, UFS caches pages in the vm object  
based on their offset in the file. Thus it would never be possible to  
use blocks in the ARC for mmap - for applications that dirty file  
backed mmaped memory it will always be necessary to have two copies of  
the page, one in the vm object for the file and one in ZFS that maps  
to a block offset. It all makes a bit more sense if you understand  
that ZFS is a transactional object store with a posix file system  
interface.