recommended memory for zfs

[Jeremy Chadwick]

On Thu, May 09, 2013 at 09:47:27PM -0400, Benjamin Adams wrote:
> On 05/09/2013 08:53 PM, Shane Ambler wrote:
> >On 09/05/2013 22:48, Benjamin Adams wrote:
> >>Hello zfs question about memory.
> >>I heard zfs is very ram hungry.
> >>Service looking to run:
> >>- nginx
> >>- postgres
> >>- php-fpm
> >>- python
> >>
> >>I have a machine with two quad core cpus but only 4 G Memory
> >>
> >>I'm looking to buy more ram now.
> >>What would be the recommend amount of memory for zfs across 6 drives on
> >>this setup?
> >>
> >
> >I believe I heard a calculation of 1GB cache per 1TB of disk. But
> >basically zfs will use all free ram available if you access that
> >much data from disk. You will want to set vfs.zfs.arc_max to allow
> >enough ram for your apps to work in.
> >
> >If you consider the files for your website and the data you store
> >you may find that you would never fill more than 500MB of cache.
> >
> >If you will be serving large media files that will easily use up
> >the cache you could give them their own filesystem that only
> >caches metadata - zfs set primarycache=metadata zroot/mediafiles
> >
> >
> Thanks for all the replies  Size of DB and HD's are:
> 
> Current DB Size = 23 GB
> HD sizes = (6) 500 GB drives

Nobody is going to be able to give you a precise/accurate recommendation
given the lack of detail provided, I'm sorry to say.  What's the RES
size of nginx (all processes combined)?  What's the RES size of
postgres (same)?  Do you have PHP scripts that "run amok" for long
periods of time and take up lots of RAM?  Same with python?  How many
concurrent visitors and what sort of content are you hosting?  Do you
maintain/write your own PHP/Python code or are you using some crap like
Wordpress?

This is just a **small** list of questions -- and what may come as a
shock is that I do not expect you to provide answers to any of them.
They are questions that you should, for yourself, attempt to answer and
work out what you need from there ("teach a man to fish" and all that).

The advice of "1GB of RAM per 1TB of disk space" is absolute nonsense on
numerous levels -- whoever gave this advice to Shane either has no
understanding of how filesystems/ZFS works, or does but chose to
simplify to the point where they're providing half-ass information.
There is no direct, or even indirect, correlation between disk capacity
and ZFS ARC size -- what matter is your "working set" (to quote Tom).
You need to have some idea of how much disk I/O you're doing, and what
type of I/O (sequential or random).

If you want my general advice, Benjamin, it's this: get yourself a
system with *minimum* 8GB of RAM but has the physical possibility of
supporting more (and only add more RAM when/if you know you need it); do
not bother with ZFS on a system with 4GB.  Run amd64, not i386 (I don't
recommend bothering with ZFS on i386 -- I am not going to get into a
discussion about this either).  Run stable/9, not 9.1-RELEASE.  Avoid
compression and dedup.  And test disk failures as well (don't get caught
with your pants down later).

The above advice comes from someone who did hosting (web/ssh/etc.) for
almost 20 years with KISS principle applied at all levels.  YMMV though,
depending on what all you're doing/what you truly need.

Good luck.

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |