Improving ZFS performance for large directories

[Peter Jeremy]

On 2013-Jan-29 18:06:01 -0600, Kevin Day <toasty at dragondata.com> wrote:
>On Jan 29, 2013, at 5:42 PM, Matthew Ahrens <mahrens at delphix.com> wrote:
>> On Tue, Jan 29, 2013 at 3:20 PM, Kevin Day <toasty at dragondata.com> wrote:
>> I'm prepared to try an L2arc cache device (with secondarycache=metadata),
>> 
>> You might first see how long it takes when everything is cached.  E.g. by doing this in the same directory several times.  This will give you a lower bound on the time it will take (or put another way, an upper bound on the improvement available from a cache device).
>>  
>
>Doing it twice back-to-back makes a bit of difference but it's still slow either way.

ZFS can very conservative about caching data and twice might not be enough.
I suggest you try 8-10 times, or until the time stops reducing.

>I think some of the issue is that nothing is being allowed to stay cached long.

Well ZFS doesn't do any time-based eviction so if things aren't
staying in the cache, it's because they are being evicted by things
that ZFS considers more deserving.

Looking at the zfs-stats you posted, it looks like your workload has
very low locality of reference (the data hitrate is very) low.  If
this is not what you expect then you need more RAM.  OTOH, your
vfs.zfs.arc_meta_used being above vfs.zfs.arc_meta_limit suggests that
ZFS really wants to cache more metadata (by default ZFS has a 25%
metadata, 75% data split in ARC to prevent metadata caching starving
data caching).  I would go even further than the 50:50 split suggested
later and try 75:25 (ie, triple the current vfs.zfs.arc_meta_limit).

Note that if there is basically no locality of reference in your
workload (as I suspect), you can even turn off data caching for
specific filesystems with zfs set primarycache=metadata tank/foo
(note that you still need to increase vfs.zfs.arc_meta_limit to
allow ZFS to use the the ARC to cache metadata).

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20130202/43d98b1d/attachment.sig>