zfs arc - just take it all and be good to me

[Marco van Tol]

On Wed, Aug 11, 2010 at 01:08:51PM -0700, Artem Belevich wrote:
> > The bottom line of my original mail was:
> > - Hey, I set the arc to just take all my RAM
> > - It grows on reads and writes
> > - It shrinks when programs need the memory it has.
> > - It behaves well in a scenario of combined read/write/other programs.
> > - This is good, really good, what is dangerous about my settings that
> >  not everybody just sets the arc to nearly all their physical memory on
> >  FreeBSD?
> 
> The issue is that ARC will give up memory even if there's plenty of it
> available and sitting in inactive/cache queues. That's particularly
> nasty in case your system uses some other filesystem besides ZFS. For
> example try tarring up few gigabytes worth of data from UFS filesystem
> and see how far your ARC size shrinks. It could be mitigated by
> setting minimum ARC size to be large enough so ZFS performance does
> not degrade. The downside is that ARC will not give up memory below
> its minimum no matter what, so if your APP really needs it, it would
> have to go to swap.
> 
> There's a hack floating around that attempts to force kernel into
> freeing up memory from inactive/cache lists before draining ARC. It
> does help a bit with this issue, but it's still a hack.

That makes sense Artem, thanks.  I think you mean the posts with the
perl one-liner I used in my tests as well.  (perl variable assignment of
1.5GB in the posts their version)
I had seen the posts that mentioned that one and decided to remember the
perl hack. :)

What I understand from it:
- In a UFS/ZFS mixed system
- In a scenario where UFS "page cache" took (almost) all available memory
- Run a perl one-liner to throw out the UFS active/inactive usage
- Kind-off hope you do enough relevant ZFS accesses that you get a good
  new situation.

So, if my worries can shift from fighting with kmem_size and arc_max to
fighting with arc_min, that's a fight I like a lot better.  Especially
on zfs-only systems, I have to admit.

Thanks a lot!

Marco

-- 
It's fried rice, you plick. -- Lethal Weapon 4