zfs very poor performance compared to ufs due to lack of cache?

[Andriy Gapon]

on 21/09/2010 13:55 Steven Hartland said the following:
> 
> ----- Original Message ----- From: "Andriy Gapon" <avg at freebsd.org>
> To: "Steven Hartland" <killing at multiplay.co.uk>
> Cc: <freebsd-fs at freebsd.org>
> Sent: Tuesday, September 21, 2010 10:51 AM
> Subject: Re: zfs very poor performance compared to ufs due to lack of cache?
> 
> 
>> on 21/09/2010 12:49 Andriy Gapon said the following:
>>> on 21/09/2010 12:27 Steven Hartland said the following:
>>>> forces into Inact and ARC never seems to push back to balance this out :(
>>>
>>> Just a general note here.
>>> ARC is not designed to "push back".  It's designed to give up memory under
>>> pressure, it's designed to expand into free space, it's not designed to create
>>> the pressure.
>>> Incorrect language produces incorrect perception resulting in incorrect
>>> expectations.
> 
> It was my understanding that ARC when looking for available memory took Inactive
> pages into account?

In what sense?
Inactive is not free.  Inactive pages are as used as ARC pages.

>> I guess what I wanted to say is - why do you want ARC to grow more in this case?
>> When you know that the data that sendfile uses is in those Inactive pages.
> 
> Well my understanding thus far, correct me if this is wrong, is that unless the data
> in the memory populated by the use of sendfile "matches" those in ARC, having it
> occupy that memory is pointless as it will never get used?

I don't follow your use of word "matches".
As long as data populated by sendfile is in memory (using your terminology)
sendfile is able to re-use it without additional trips to ARC or to disk.

> If this is the case all you will see is a cycling of memory use for different
> files and the
> overall result will be that only ever 1.5G of files are actually cached,
> everything else
> must still be read from disk "and" copied in memory before being sent?

Inactive is also a cache.

> What I would expect is the natural balance to be achieved, with Inactive pages and
> ARC having an pretty even split. So on this machine with 7G say 500M possibly 1G
> general overhead at a real push it would result in 3G Inactive pages from sendfile
> matching the 3G of ARC?
> 
> ATM the only way to achieve this seems to be to manually tune min arc setting to this
> level?
> 
> Sorry this is all questions, as I really don't have a clear picture of how this
> all works :(

Yes, you really need to understand how VM works first.
Think of "what sendfile populates" as L1 cache and ARC as L2 cache with inclusive
relation-ship (i.e. the same data can be in both).  The differences from CPUs is
that balance of sizes between L1 and L2 is established dynamically.  Another
difference is that some operations like read(2) bypass L1 and go to L2 directly.
If you use operations that work through L1 and most of your data is already in L1,
then why you'd want L2 to be large?

-- 
Andriy Gapon