ZFS and large directories - caveat report

[Ivan Voras]

On 21 July 2011 17:50, Freddie Cash <fjwcash at gmail.com> wrote:
> On Thu, Jul 21, 2011 at 8:45 AM, Ivan Voras <ivoras at freebsd.org> wrote:
>>
>> Is there an equivalent of UFS dirhash memory setting for ZFS? (i.e. the
>> size of the metadata cache)
>
> vfs.zfs.arc_meta_limit
>
> This sets the amount of ARC that can be used for metadata.  The default is
> 1/8th of ARC, I believe.  This setting lets you use "primarycache=all"
> (store metadata and file data in ARC) but then tune how much is used for
> each.
>
> Not sure if that will help in your case or not, but it's a sysctl you can
> play with.

I don't think that it works, or at least is not as efficient as dirhash:

www:~> sysctl -a | grep meta
kern.metadelay: 28
vfs.zfs.mfu_ghost_metadata_lsize: 129082368
vfs.zfs.mfu_metadata_lsize: 116224
vfs.zfs.mru_ghost_metadata_lsize: 113958912
vfs.zfs.mru_metadata_lsize: 16384
vfs.zfs.anon_metadata_lsize: 0
vfs.zfs.arc_meta_limit: 322412800
vfs.zfs.arc_meta_used: 506907792
kstat.zfs.misc.arcstats.demand_metadata_hits: 4471705
kstat.zfs.misc.arcstats.demand_metadata_misses: 2110328
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 27
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 51

arc_meta_used is nearly 500 MB which should be enough even in this
case. With filenames of 32 characters, all the filenames alone for
130,000 files in a directory take about 4 MB - I doubt the ZFS
introduces so much extra metadata it doesn't fit in 500 MB.

I am now deleting the session files, and I hope it will not take days
to complete...