recommended memory for zfs

Fri May 10 10:48:45 UTC 2013

On May 10, 2013, at 1:08 PM, "Ronald Klop" <ronald-freebsd8 at klop.yi.org> wrote:

> On Fri, 10 May 2013 04:18:42 +0200, Benjamin Adams <benjamindadams at gmail.com> wrote:
> 
>> On 05/09/2013 10:06 PM, Jeremy Chadwick wrote:
>>> On Thu, May 09, 2013 at 09:47:27PM -0400, Benjamin Adams wrote:
>>>> On 05/09/2013 08:53 PM, Shane Ambler wrote:
>>>>> On 09/05/2013 22:48, Benjamin Adams wrote:
>>>>>> Hello zfs question about memory.
>>>>>> I heard zfs is very ram hungry.
>>>>>> Service looking to run:
>>>>>> - nginx
>>>>>> - postgres
>>>>>> - php-fpm
>>>>>> - python
>>>>>> 
>>>>>> I have a machine with two quad core cpus but only 4 G Memory
>>>>>> 
>>>>>> I'm looking to buy more ram now.
>>>>>> What would be the recommend amount of memory for zfs across 6 drives on
>>>>>> this setup?
>>>>>> 
>>>>> I believe I heard a calculation of 1GB cache per 1TB of disk. But
>>>>> basically zfs will use all free ram available if you access that
>>>>> much data from disk. You will want to set vfs.zfs.arc_max to allow
>>>>> enough ram for your apps to work in.
>>>>> 
>>>>> If you consider the files for your website and the data you store
>>>>> you may find that you would never fill more than 500MB of cache.
>>>>> 
>>>>> If you will be serving large media files that will easily use up
>>>>> the cache you could give them their own filesystem that only
>>>>> caches metadata - zfs set primarycache=metadata zroot/mediafiles
>>>>> 
>>>>> 
>>>> Thanks for all the replies  Size of DB and HD's are:
>>>> 
>>>> Current DB Size = 23 GB
>>>> HD sizes = (6) 500 GB drives
>>> Nobody is going to be able to give you a precise/accurate recommendation
>>> given the lack of detail provided, I'm sorry to say.  What's the RES
>>> size of nginx (all processes combined)?  What's the RES size of
>>> postgres (same)?  Do you have PHP scripts that "run amok" for long
>>> periods of time and take up lots of RAM?  Same with python?  How many
>>> concurrent visitors and what sort of content are you hosting?  Do you
>>> maintain/write your own PHP/Python code or are you using some crap like
>>> Wordpress?
>>> 
>>> This is just a **small** list of questions -- and what may come as a
>>> shock is that I do not expect you to provide answers to any of them.
>>> They are questions that you should, for yourself, attempt to answer and
>>> work out what you need from there ("teach a man to fish" and all that).
>>> 
>>> The advice of "1GB of RAM per 1TB of disk space" is absolute nonsense on
>>> numerous levels -- whoever gave this advice to Shane either has no
>>> understanding of how filesystems/ZFS works, or does but chose to
>>> simplify to the point where they're providing half-ass information.
>>> There is no direct, or even indirect, correlation between disk capacity
>>> and ZFS ARC size -- what matter is your "working set" (to quote Tom).
>>> You need to have some idea of how much disk I/O you're doing, and what
>>> type of I/O (sequential or random).
>>> 
>>> If you want my general advice, Benjamin, it's this: get yourself a
>>> system with *minimum* 8GB of RAM but has the physical possibility of
>>> supporting more (and only add more RAM when/if you know you need it); do
>>> not bother with ZFS on a system with 4GB.  Run amd64, not i386 (I don't
>>> recommend bothering with ZFS on i386 -- I am not going to get into a
>>> discussion about this either).  Run stable/9, not 9.1-RELEASE.  Avoid
>>> compression and dedup.  And test disk failures as well (don't get caught
>>> with your pants down later).
>>> 
>>> The above advice comes from someone who did hosting (web/ssh/etc.) for
>>> almost 20 years with KISS principle applied at all levels.  YMMV though,
>>> depending on what all you're doing/what you truly need.
>>> 
>>> Good luck.
>>> 
>> Jeremy,
>> 
>> Was just see if I should just get raid controller and more ram down the road.
>> List of priorities.
>> 
>> Main thing is I move from BSD when 9.0 came out.  Was looking to see if zfs is included in the installer.  Now.
>> 
>> Sum up:
>> upgrade ram to 16GB (not 64 like plained)
>> and  raid controller that supports level 5.
>> 
> 
> Let ZFS do the RAID stuff. Do not use a RAID controller, but give the plain disks to ZFS. Some of the nice features come from ZFS doing the RAID stuff.

To paraphrase this.

Get yourself a nice HBA. non-RAID! For example, something based on LSI2008 with IT firmware. With few enough discs you can avoid using SAS expanders as well. RAID controllers, in addition to causing all kinds of troubles are unlikely to have sufficient bandwidth and might turn out to be the bottleneck (unless you are prepared to spend an unholy amount of money -- which are better spent for RAM and CPU).

If you want performance and low latency, avoid using compression and dedup in ZFS. Set your record size appropriately for postgresql (8k) *before* you run initdb. It is best to create an separate filesystem for the database and set that property only there. If your database is heavy on updates, you might be interested to use an SSD for ZIL. In general, if you can afford it, an cheap SSD for L2ARC might do wonders -- if your data set can fit there. If you intend to use SSDs and want the best performance, use different SSDs for ZIL and L2ARC. The first needs to be fast at writing and optimised for his (for example, the OCZ Vector) -- it does not need to be large, at all. The second has to provide high read throughput and IOPS, not so much for writing.

Lately, I am building more and more servers with SSDs only and ZFS and performance has been incomparable with spinning drives.

Also, check what the sector size on your drives is. Most these days are 4k already and you need to tell ZFS that (because the drives lie). It is safer to plan for 4k drives, as future replacements are likely to be of that kind. The same about SSDs. 

Daniel