zfs very poor performance compared to ufs due to lack of cache?

[Andriy Gapon]

on 15/09/2010 13:32 Steven Hartland said the following:
> === conclusion ===
> The interaction of zfs and sendfile is causing large amounts of memory
> to end up in the inactive pool and only the use of a hard min arc limit is
> ensuring that zfs forces the vm to release said memory so that it can be
> used by zfs arc.

Memory ends up as inactive because of how sendfile works.  It first pulls data
into a page cache as active pages.  After pages are not used for a while, they
become inactive.  Pagedaemon can further recycle inactive pages, but only if
there is any shortage.  In your situation there is no shortage, so pages just
stay there, but are ready to be reclaimed (or re-activated) at any moment.
They are not a waste!  Just a form of a cache.
If ARC size doesn't grow in that condition, then it means that ZFS simply
doesn't need it to.

General problem of double-caching with ZFS still remains and will remain and
nobody promised to fix that.
I.e. with sendfile (or mmap) you will end up with two copies of data, one in
page cache and the other in ARC.  That happens on Solaris too, no magic.

The things I am trying to fix are:
1. Interaction between ARC and the rest of VM during page shortage; you don't
seem to have much of that, so you don't see it.  Besides, your range for ARC
size is quite narrow and your workload is so peculiar that your setup is not the
best one for testing this.
2. Copying of data from ARC to page cache each time the same data is served by
sendfile.  You won't see much changes without monitoring ARC hits as Wiktor has
suggested.  In bad case there would be many hits because the same data is
constantly copied from ARC to page cache (and that simply kills any benefit
sendfile may have).  In good case there would be much less hits, because data is
not copied, but is served directly from page cache.

> The source data, xls's and exported graphs can be found here:-
> http://www.multiplaygameservers.com/dropzone/zfs-sendfile-results.zip

So, what problem, performance or otherwise, do you perceive with your system's
behavior?  Because I don't see any.

To summarize:
1. With sendfile enabled you will have two copies of actively served data in
RAM, but perhaps slightly faster performance, because of avoiding another copy
to mbuf in sendfile(2).
2. With sendfile disabled, you will have one copy of actively served data in RAM
(in ARC), but perhaps slightly slower performance because of a need to make a
copy to mbuf.

Which would serve you better depends on size of your hot data vs RAM size, and
on actual benefit from avoiding the copying to mbuf.  I have never measured the
latter, so I don't have any real numbers.
>From your graphs it seems that your hot data (multiplied by two) is larger than
what your RAM can accommodate, so you should benefit from disabling sendfile.

-- 
Andriy Gapon