zfs very poor performance compared to ufs due to lack of cache?

Steven Hartland killing at multiplay.co.uk
Wed Sep 15 20:00:43 UTC 2010


----- Original Message ----- 
From: "Andriy Gapon" <avg at freebsd.org>
> Or even try the opposite, if your version of ZFS permits it.
> You can set primarycache=metadata on the filesystem where you have the data that
> you serve via sendfile.  With that setting it shouldn't get cached in ARC, but it
> should be still cached in VM cache, so you should get UFS-like behavior.
> 
> Will you test it? :)

Ok given this a whirl, don't have the full results just yet but does seem that buf cache is not
used at all?

Mem: 32M Active, 1378M Inact, 159M Wired, 120K Cache, 21M Buf, 5348M Free
Swap: 4096M Total, 4096M Free

It also appears to have totally destroyed overall disk IO performance, particularly for small
reads e.g. those used in cat.

I just tried to cat a 1.8GB file from disk to /dev/null which I would usually expect to ~170MB/s
on initial read but I only to got 1.8MB/s even though the disk subsystem was doing 200MB/s
to sustain this request. Massive over read which is then just thrown away because there is
no data cache?

This seems backed up by using dd with a block size over 128K (zfs block size I believe?)
results in normal performance of 180MB/s but repeat runs still only get the same, where on
a similar box with ufs I see 1.3GB/s on repeat runs.

Going to pull the test there as the machines struggling to keep up with even 10 clients.

Next test I think should be sendfile on but with no special zfs loader.conf options to see what
really happens to arc without any limits. I've got a suspicion that I may end up with close to
or zero arc due to inact memory use.

    Regards
    Steve

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. 

In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster at multiplay.co.uk.



More information about the freebsd-fs mailing list