Another ZFS ARC memory question

[Peter Jeremy]

[Cc list pruned]

On 2012-Mar-02 10:16:06 +0000, Luke Marsden <luke-lists at hybrid-logic.co.uk> wrote:
>We are presently working around it by limiting arc_max to 4G on our 24G
>RAM production boxes (which seems like a massive waste of performance)
>and by doing very careful/aggressive application level management of
>memory usage to ensure stability (limits.conf didn't work for us, so we
>rolled our own).  A better solution would be welcome, though, so that we
>can utilise all the free memory we're presently keeping around as a
>safety margin - ideally it would be used as ARC cache.

Have you tried increasing vm.v_cache_min to cover your spikes?

>1.  Is it expected that with a 4G limited arc_max value that we should
>see wired memory usage around 7-8G?  I understand that the kernel has to
>use some memory, but really 3-4G of non-ARC data?

Yes, that sounds possible.

>2.  We have some development machines with only 3G of RAM.  Previously
>they had no arc_max set and were left to tune themselves.  They were
>quite unstable.  Now we've set arc_max to 256M but things have got
>worse: we've seen a disk i/o big performance hit (untarring a ports
>tarball now takes 20 minutes), and wired memory usage is up around
>2.5GB, the machines are swapping a lot, and crashing more frequently.

That's stress-testing ZFS more than anything else.  You definitely
can't use those results as a guide to tune your production boxes
(other than what not to do).  That said, I have 3.5GB in my $work
desktop (running 8.2-stable from about a month ago) and don't have
any stability issues with either it or a buildbox with 2GB RAM.

>Follows is arc_summary.pl from one of the troubled dev machines showing
>the ARC using 500% of the memory it should be.  Also uname follows.  My
>second question is: have there been fixes between 8.2-RELEASE and
>8.3-BETA1 or 9.0-RELEASE which solve this ARC over-usage problem?

There definitely have been some commits to ensure that arc_max is
treated much more as a hard limit but I can't quickly find them so
I'm not sure if they pre- or post-date 8.2-RELEASE.

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20120302/3c999852/attachment.pgp