ZFS and large directories - caveat report

[Attila Nagy]

On 07/21/11 18:38, Ivan Voras wrote:
> On 21 July 2011 17:50, Freddie Cash<fjwcash at gmail.com>  wrote:
>> On Thu, Jul 21, 2011 at 8:45 AM, Ivan Voras<ivoras at freebsd.org>  wrote:
>>> Is there an equivalent of UFS dirhash memory setting for ZFS? (i.e. the
>>> size of the metadata cache)
>> vfs.zfs.arc_meta_limit
>>
>> This sets the amount of ARC that can be used for metadata.  The default is
>> 1/8th of ARC, I believe.  This setting lets you use "primarycache=all"
>> (store metadata and file data in ARC) but then tune how much is used for
>> each.
>>
>> Not sure if that will help in your case or not, but it's a sysctl you can
>> play with.
> I don't think that it works, or at least is not as efficient as dirhash:
>
> www:~>  sysctl -a | grep meta
> kern.metadelay: 28
> vfs.zfs.mfu_ghost_metadata_lsize: 129082368
> vfs.zfs.mfu_metadata_lsize: 116224
> vfs.zfs.mru_ghost_metadata_lsize: 113958912
> vfs.zfs.mru_metadata_lsize: 16384
> vfs.zfs.anon_metadata_lsize: 0
> vfs.zfs.arc_meta_limit: 322412800
> vfs.zfs.arc_meta_used: 506907792
> kstat.zfs.misc.arcstats.demand_metadata_hits: 4471705
> kstat.zfs.misc.arcstats.demand_metadata_misses: 2110328
> kstat.zfs.misc.arcstats.prefetch_metadata_hits: 27
> kstat.zfs.misc.arcstats.prefetch_metadata_misses: 51
>
> arc_meta_used is nearly 500 MB which should be enough even in this
> case. With filenames of 32 characters, all the filenames alone for
> 130,000 files in a directory take about 4 MB - I doubt the ZFS
> introduces so much extra metadata it doesn't fit in 500 MB.
>
> I am now deleting the session files, and I hope it will not take days
> to complete...
>
Worse than that, I've seen a similar issue, hashed directories with 
about 1M+ files. After deleting all those files, even a find on the 
empty directories took ages...