ARC size constantly shrinks, then ZFS slows down extremely

[K. Macy]

>
> The amd64 machine started to loose ARC memory again. See these:
> http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/zfs_mem- 
> week.png
> http://people.fsn.hu/~bra/freebsd/20091012-zfs-arcsize/memory-week.png
>
> Your patch was active between 7 and 9. You can see that the ARC size  
> was somewhat constant.
> On october 9, I installed Kip's modification, and ARC size started  
> to decrease.
> BTW, previously (before october 7) I set the arc min size to 10-15GB  
> (can't remember the exact value), but now it runs with the defaults  
> (only the max size is set):
> vfs.zfs.arc_min: 3623878656
> vfs.zfs.arc_max: 28991029248
>
> As you can see, there are plenty of memory. This machine uses UFS as  
> well (and writes it heavily), maybe that's what affects ZFS size, by  
> caching a lot of stuff?
>

Currently, the inactive page queue will grow until ARC is shrunk to  
arc_min.


I think I'll probably spend some time making the ARC play better with  
the page cache this week. Unfortunately, under heavy memory pressure  
when competing with UFS the ARC will degrade to LRU, but I think that  
is still an improvement over the current static sizing with low and  
high water marks.

-Kip