Improving ZFS performance for large directories
Kevin Day
toasty at dragondata.com
Wed Jan 30 15:19:59 UTC 2013
On Jan 30, 2013, at 4:36 AM, Nikolay Denev <ndenev at gmail.com> wrote:
>
>
> What are your : vfs.zfs.arc_meta_limit and vfs.zfs.arc_meta_used sysctls?
> Maybe increasing the limit can help?
vfs.zfs.arc_meta_limit: 8199079936
vfs.zfs.arc_meta_used: 13965744408
Full output of zfs-stats:
------------------------------------------------------------------------
ZFS Subsystem Report Wed Jan 30 15:16:54 2013
------------------------------------------------------------------------
System Information:
Kernel Version: 901000 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 28
ZFS Filesystem Version: 5
FreeBSD 9.1-RC2 #1: Tue Oct 30 20:37:38 UTC 2012 root
3:16PM up 19 days, 19:44, 2 users, load averages: 0.91, 0.80, 0.68
------------------------------------------------------------------------
System Memory:
12.44% 7.72 GiB Active, 6.04% 3.75 GiB Inact
77.33% 48.01 GiB Wired, 2.25% 1.40 GiB Cache
1.94% 1.21 GiB Free, 0.00% 1.21 MiB Gap
Real Installed: 64.00 GiB
Real Available: 99.97% 63.98 GiB
Real Managed: 97.04% 62.08 GiB
Logical Total: 64.00 GiB
Logical Used: 90.07% 57.65 GiB
Logical Free: 9.93% 6.35 GiB
Kernel Memory: 22.62 GiB
Data: 99.91% 22.60 GiB
Text: 0.09% 21.27 MiB
Kernel Memory Map: 54.28 GiB
Size: 34.75% 18.86 GiB
Free: 65.25% 35.42 GiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 430.91m
Recycle Misses: 111.27m
Mutex Misses: 2.49m
Evict Skips: 647.25m
ARC Size: 87.63% 26.77 GiB
Target Size: (Adaptive) 87.64% 26.77 GiB
Min Size (Hard Limit): 12.50% 3.82 GiB
Max Size (High Water): 8:1 30.54 GiB
ARC Size Breakdown:
Recently Used Cache Size: 58.64% 15.70 GiB
Frequently Used Cache Size: 41.36% 11.07 GiB
ARC Hash Breakdown:
Elements Max: 2.19m
Elements Current: 86.15% 1.89m
Collisions: 344.47m
Chain Max: 17
Chains: 552.47k
------------------------------------------------------------------------
ARC Efficiency: 21.94b
Cache Hit Ratio: 97.00% 21.28b
Cache Miss Ratio: 3.00% 657.23m
Actual Hit Ratio: 73.15% 16.05b
Data Demand Efficiency: 98.94% 1.32b
Data Prefetch Efficiency: 14.83% 299.44m
CACHE HITS BY CACHE LIST:
Anonymously Used: 23.03% 4.90b
Most Recently Used: 6.12% 1.30b
Most Frequently Used: 69.29% 14.75b
Most Recently Used Ghost: 0.50% 105.94m
Most Frequently Used Ghost: 1.07% 226.92m
CACHE HITS BY DATA TYPE:
Demand Data: 6.11% 1.30b
Prefetch Data: 0.21% 44.42m
Demand Metadata: 69.29% 14.75b
Prefetch Metadata: 24.38% 5.19b
CACHE MISSES BY DATA TYPE:
Demand Data: 2.12% 13.90m
Prefetch Data: 38.80% 255.02m
Demand Metadata: 30.97% 203.56m
Prefetch Metadata: 28.11% 184.75m
------------------------------------------------------------------------
L2ARC is disabled
------------------------------------------------------------------------
File-Level Prefetch: (HEALTHY)
DMU Efficiency: 24.08b
Hit Ratio: 66.02% 15.90b
Miss Ratio: 33.98% 8.18b
Colinear: 8.18b
Hit Ratio: 0.01% 560.82k
Miss Ratio: 99.99% 8.18b
Stride: 15.23b
Hit Ratio: 99.98% 15.23b
Miss Ratio: 0.02% 2.62m
DMU Misc:
Reclaim: 8.18b
Successes: 0.08% 6.31m
Failures: 99.92% 8.17b
Streams: 663.44m
+Resets: 0.06% 397.18k
-Resets: 99.94% 663.04m
Bogus: 0
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 384
vm.kmem_size 66662760448
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 329853485875
vfs.zfs.l2c_only_size 0
vfs.zfs.mfu_ghost_data_lsize 2121007104
vfs.zfs.mfu_ghost_metadata_lsize 7876605440
vfs.zfs.mfu_ghost_size 9997612544
vfs.zfs.mfu_data_lsize 10160539648
vfs.zfs.mfu_metadata_lsize 17161216
vfs.zfs.mfu_size 11163991040
vfs.zfs.mru_ghost_data_lsize 7235079680
vfs.zfs.mru_ghost_metadata_lsize 11107812352
vfs.zfs.mru_ghost_size 18342892032
vfs.zfs.mru_data_lsize 4406255616
vfs.zfs.mru_metadata_lsize 3924364288
vfs.zfs.mru_size 8893582336
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 999424
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 8199079936
vfs.zfs.arc_meta_used 14161977912
vfs.zfs.arc_min 4099539968
vfs.zfs.arc_max 32796319744
vfs.zfs.dedup.prefetch 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.write_limit_override 0
vfs.zfs.write_limit_inflated 206088929280
vfs.zfs.write_limit_max 8587038720
vfs.zfs.write_limit_min 33554432
vfs.zfs.write_limit_shift 3
vfs.zfs.no_write_throttle 0
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 0
vfs.zfs.mg_alloc_failures 12
vfs.zfs.check_hostid 1
vfs.zfs.recover 0
vfs.zfs.txg.synctime_ms 1000
vfs.zfs.txg.timeout 5
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.ramp_rate 2
vfs.zfs.vdev.time_shift 6
vfs.zfs.vdev.min_pending 4
vfs.zfs.vdev.max_pending 10
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_replay_disable 0
vfs.zfs.zio.use_uma 0
vfs.zfs.snapshot_list_prefetch 0
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 28
vfs.zfs.version.acl 1
vfs.zfs.debug 0
vfs.zfs.super_owner 0
------------------------------------------------------------------------
More information about the freebsd-fs
mailing list