ZFS directory with a large number of files

[seanrees at gmail.com]

On Tue, Aug 2, 2011 at 12:07 PM, Jeremy Chadwick
<freebsd at jdc.parodius.com> wrote:
> On Tue, Aug 02, 2011 at 11:55:43AM +0100, seanrees at gmail.com wrote:
>> On Tue, Aug 2, 2011 at 11:10 AM, Daniel Kalchev <daniel at digsys.bg> wrote:
>> >> If it is a limitation in ZFS it would be nice to know that, perhaps it
>> >> truly, really is a bug that can be avoided (or it's inherent in the way ZFS
>> >> handles such things)
>> >
>> > It is possible ?that there is not enough memory in ARC to cache that large
>> > directory.
>> >
>> > Other than that, perhaps in ZFS it would be easier to prune the unused
>> > directory entries, than it is in UFS. It looks like this is not implemented.
>> >
>> > Another reason might be some FreeBSD specific implementation issue for
>> > fstatfs.
>> >
>> > In any case, the data available is not sufficient. More information would
>> > help, like how much RAM this system has, how much ARC uses, some ARC stats.
>>
>> Which sysctl's would you like?
>
> Output from "sysctl vfs.zfs kstat.zfs" would be sufficient.

Here we are:


vfs.zfs.l2c_only_size: 0
vfs.zfs.mfu_ghost_data_lsize: 0
vfs.zfs.mfu_ghost_metadata_lsize: 26383360
vfs.zfs.mfu_ghost_size: 26383360
vfs.zfs.mfu_data_lsize: 0
vfs.zfs.mfu_metadata_lsize: 154112
vfs.zfs.mfu_size: 3944960
vfs.zfs.mru_ghost_data_lsize: 0
vfs.zfs.mru_ghost_metadata_lsize: 76250624
vfs.zfs.mru_ghost_size: 76250624
vfs.zfs.mru_data_lsize: 30208
vfs.zfs.mru_metadata_lsize: 16896
vfs.zfs.mru_size: 29353984
vfs.zfs.anon_data_lsize: 0
vfs.zfs.anon_metadata_lsize: 0
vfs.zfs.anon_size: 150016
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.arc_meta_limit: 26214400
vfs.zfs.arc_meta_used: 108539456
vfs.zfs.arc_min: 33554432
vfs.zfs.arc_max: 104857600
vfs.zfs.dedup.prefetch: 1
vfs.zfs.mdcomp_disable: 0
vfs.zfs.write_limit_override: 0
vfs.zfs.write_limit_inflated: 6360993792
vfs.zfs.write_limit_max: 265041408
vfs.zfs.write_limit_min: 33554432
vfs.zfs.write_limit_shift: 3
vfs.zfs.no_write_throttle: 0
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.block_cap: 256
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 1
vfs.zfs.check_hostid: 1
vfs.zfs.recover: 0
vfs.zfs.txg.synctime_ms: 1000
vfs.zfs.txg.timeout: 5
vfs.zfs.scrub_limit: 10
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 10485760
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.write_gap_limit: 4096
vfs.zfs.vdev.read_gap_limit: 32768
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.ramp_rate: 2
vfs.zfs.vdev.time_shift: 6
vfs.zfs.vdev.min_pending: 4
vfs.zfs.vdev.max_pending: 10
vfs.zfs.vdev.bio_flush_disable: 0
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_replay_disable: 0
vfs.zfs.zio.use_uma: 0
vfs.zfs.version.zpl: 5
vfs.zfs.version.spa: 28
vfs.zfs.version.acl: 1
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
kstat.zfs.misc.xuio_stats.onloan_read_buf: 0
kstat.zfs.misc.xuio_stats.onloan_write_buf: 0
kstat.zfs.misc.xuio_stats.read_buf_copied: 0
kstat.zfs.misc.xuio_stats.read_buf_nocopy: 0
kstat.zfs.misc.xuio_stats.write_buf_copied: 0
kstat.zfs.misc.xuio_stats.write_buf_nocopy: 107064
kstat.zfs.misc.zfetchstats.hits: 0
kstat.zfs.misc.zfetchstats.misses: 0
kstat.zfs.misc.zfetchstats.colinear_hits: 0
kstat.zfs.misc.zfetchstats.colinear_misses: 0
kstat.zfs.misc.zfetchstats.stride_hits: 0
kstat.zfs.misc.zfetchstats.stride_misses: 0
kstat.zfs.misc.zfetchstats.reclaim_successes: 0
kstat.zfs.misc.zfetchstats.reclaim_failures: 0
kstat.zfs.misc.zfetchstats.streams_resets: 0
kstat.zfs.misc.zfetchstats.streams_noresets: 0
kstat.zfs.misc.zfetchstats.bogus_streams: 0
kstat.zfs.misc.arcstats.hits: 47091548
kstat.zfs.misc.arcstats.misses: 17064059
kstat.zfs.misc.arcstats.demand_data_hits: 15357194
kstat.zfs.misc.arcstats.demand_data_misses: 3077290
kstat.zfs.misc.arcstats.demand_metadata_hits: 31102404
kstat.zfs.misc.arcstats.demand_metadata_misses: 8692242
kstat.zfs.misc.arcstats.prefetch_data_hits: 0
kstat.zfs.misc.arcstats.prefetch_data_misses: 0
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 631950
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 5294527
kstat.zfs.misc.arcstats.mru_hits: 27566971
kstat.zfs.misc.arcstats.mru_ghost_hits: 2179308
kstat.zfs.misc.arcstats.mfu_hits: 18950663
kstat.zfs.misc.arcstats.mfu_ghost_hits: 2714218
kstat.zfs.misc.arcstats.allocated: 19825272
kstat.zfs.misc.arcstats.deleted: 12619489
kstat.zfs.misc.arcstats.stolen: 9003539
kstat.zfs.misc.arcstats.recycle_miss: 10224598
kstat.zfs.misc.arcstats.mutex_miss: 1984
kstat.zfs.misc.arcstats.evict_skip: 216358592
kstat.zfs.misc.arcstats.evict_l2_cached: 0
kstat.zfs.misc.arcstats.evict_l2_eligible: 433025541120
kstat.zfs.misc.arcstats.evict_l2_ineligible: 87633796096
kstat.zfs.misc.arcstats.hash_elements: 15988
kstat.zfs.misc.arcstats.hash_elements_max: 43365
kstat.zfs.misc.arcstats.hash_collisions: 5599202
kstat.zfs.misc.arcstats.hash_chains: 3944
kstat.zfs.misc.arcstats.hash_chain_max: 21
kstat.zfs.misc.arcstats.p: 28381184
kstat.zfs.misc.arcstats.c: 104857600
kstat.zfs.misc.arcstats.c_min: 33554432
kstat.zfs.misc.arcstats.c_max: 104857600
kstat.zfs.misc.arcstats.size: 108700736
kstat.zfs.misc.arcstats.hdr_size: 3677448
kstat.zfs.misc.arcstats.data_size: 33448960
kstat.zfs.misc.arcstats.other_size: 71574328
kstat.zfs.misc.arcstats.l2_hits: 0
kstat.zfs.misc.arcstats.l2_misses: 0
kstat.zfs.misc.arcstats.l2_feeds: 0
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_read_bytes: 0
kstat.zfs.misc.arcstats.l2_write_bytes: 0
kstat.zfs.misc.arcstats.l2_writes_sent: 0
kstat.zfs.misc.arcstats.l2_writes_done: 0
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_hdr_miss: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_free_on_write: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_size: 0
kstat.zfs.misc.arcstats.l2_hdr_size: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 521
kstat.zfs.misc.arcstats.l2_write_trylock_fail: 0
kstat.zfs.misc.arcstats.l2_write_passed_headroom: 0
kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 0
kstat.zfs.misc.arcstats.l2_write_in_l2: 0
kstat.zfs.misc.arcstats.l2_write_io_in_progress: 0
kstat.zfs.misc.arcstats.l2_write_not_cacheable: 5352891
kstat.zfs.misc.arcstats.l2_write_full: 0
kstat.zfs.misc.arcstats.l2_write_buffer_iter: 0
kstat.zfs.misc.arcstats.l2_write_pios: 0
kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 0
kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 0
kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 0
kstat.zfs.misc.vdev_cache_stats.delegations: 5349195
kstat.zfs.misc.vdev_cache_stats.hits: 16581136
kstat.zfs.misc.vdev_cache_stats.misses: 5391734



Sean