All available memory used when deleting files from ZFS
Dustin Wenz
dustinwenz at ebureau.com
Mon Mar 30 21:24:07 UTC 2015
I had several systems panic or hang over the weekend while deleting some data off of their local zfs filesystem. It looks like they ran out of physical memory (32GB), and hung when paging to swap-on-zfs (which is not surprising, given that ZFS was likely using the memory). They were running 10.1-STABLE r277139M, which I built in the middle of January. The pools were about 35TB in size, and are a concatenation of 3TB mirrors. They were maybe 95% full. I deleted just over 1000 files, totaling 25TB on each system.
It took roughly 10 minutes to remove that 25TB of data per host using a remote rsync, and immediately after that everything seemed fine. However, after several more minutes, every machine that had data removed became unresponsive. Some had numerous "swap_pager: indefinite wait buffer" errors followed by a panic, and some just died with no console messages. The same thing would happen after a reboot, when FreeBSD attempted to mount the local filesystem again.
I was able to boot these systems after exporting the affected pool, but the problem would recur several minutes after initiating a "zpool import". Watching zfs statistics didn't seem to reveal where the memory was going; ARC would only climb to about 4GB, but free memory would decline rapidly. Eventually, after enough export/reboot/import cycles, the pool would import successfully and everything would be fine from then on. Note that there is no L2ARC or compression being used.
Has anyone else run into this when deleting files on ZFS? It seems to be a consistent problem under the versions of 10.1 I'm running.
For reference, I've appended a zstat dump below that was taken 5 minutes after starting a zpool import, and was about three minutes before the machine became unresponsive. You can see that the ARC is only 4GB, but free memory was down to 471MB (and continued to drop).
- .Dustin
------------------------------------------------------------------------
ZFS Subsystem Report Mon Mar 30 12:35:27 2015
------------------------------------------------------------------------
System Information:
Kernel Version: 1001506 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 5000
ZFS Filesystem Version: 5
FreeBSD 10.1-STABLE #11 r277139M: Tue Jan 13 14:59:55 CST 2015 root
12:35PM up 8 mins, 3 users, load averages: 7.23, 8.96, 4.87
------------------------------------------------------------------------
System Memory:
0.17% 55.40 MiB Active, 0.14% 46.11 MiB Inact
98.34% 30.56 GiB Wired, 0.00% 0 Cache
1.34% 425.46 MiB Free, 0.00% 4.00 KiB Gap
Real Installed: 32.00 GiB
Real Available: 99.82% 31.94 GiB
Real Managed: 97.29% 31.08 GiB
Logical Total: 32.00 GiB
Logical Used: 98.56% 31.54 GiB
Logical Free: 1.44% 471.57 MiB
Kernel Memory: 3.17 GiB
Data: 99.18% 3.14 GiB
Text: 0.82% 26.68 MiB
Kernel Memory Map: 31.08 GiB
Size: 14.18% 4.41 GiB
Free: 85.82% 26.67 GiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 145
Recycle Misses: 0
Mutex Misses: 0
Evict Skips: 0
ARC Size: 14.17% 4.26 GiB
Target Size: (Adaptive) 100.00% 30.08 GiB
Min Size (Hard Limit): 12.50% 3.76 GiB
Max Size (High Water): 8:1 30.08 GiB
ARC Size Breakdown:
Recently Used Cache Size: 50.00% 15.04 GiB
Frequently Used Cache Size: 50.00% 15.04 GiB
ARC Hash Breakdown:
Elements Max: 270.56k
Elements Current: 100.00% 270.56k
Collisions: 23.66k
Chain Max: 3
Chains: 8.28k
------------------------------------------------------------------------
ARC Efficiency: 2.93m
Cache Hit Ratio: 70.44% 2.06m
Cache Miss Ratio: 29.56% 866.05k
Actual Hit Ratio: 70.40% 2.06m
Data Demand Efficiency: 97.47% 24.58k
Data Prefetch Efficiency: 1.88% 479
CACHE HITS BY CACHE LIST:
Anonymously Used: 0.05% 1.07k
Most Recently Used: 71.82% 1.48m
Most Frequently Used: 28.13% 580.49k
Most Recently Used Ghost: 0.00% 0
Most Frequently Used Ghost: 0.00% 0
CACHE HITS BY DATA TYPE:
Demand Data: 1.16% 23.96k
Prefetch Data: 0.00% 9
Demand Metadata: 98.79% 2.04m
Prefetch Metadata: 0.05% 1.08k
CACHE MISSES BY DATA TYPE:
Demand Data: 0.07% 621
Prefetch Data: 0.05% 470
Demand Metadata: 99.69% 863.35k
Prefetch Metadata: 0.19% 1.61k
------------------------------------------------------------------------
L2ARC is disabled
------------------------------------------------------------------------
File-Level Prefetch: (HEALTHY)
DMU Efficiency: 72.95k
Hit Ratio: 70.83% 51.66k
Miss Ratio: 29.17% 21.28k
Colinear: 21.28k
Hit Ratio: 0.01% 2
Miss Ratio: 99.99% 21.28k
Stride: 50.45k
Hit Ratio: 99.98% 50.44k
Miss Ratio: 0.02% 9
DMU Misc:
Reclaim: 21.28k
Successes: 1.73% 368
Failures: 98.27% 20.91k
Streams: 1.23k
+Resets: 0.16% 2
-Resets: 99.84% 1.23k
Bogus: 0
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 2380
vm.kmem_size 33367830528
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 1319413950874
vfs.zfs.arc_max 32294088704
vfs.zfs.arc_min 4036761088
vfs.zfs.arc_average_blocksize 8192
vfs.zfs.arc_shrink_shift 5
vfs.zfs.arc_free_target 56518
vfs.zfs.arc_meta_used 4534349216
vfs.zfs.arc_meta_limit 8073522176
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_norw 1
vfs.zfs.anon_size 1786368
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_data_lsize 0
vfs.zfs.mru_size 504812032
vfs.zfs.mru_metadata_lsize 415273472
vfs.zfs.mru_data_lsize 35227648
vfs.zfs.mru_ghost_size 0
vfs.zfs.mru_ghost_metadata_lsize 0
vfs.zfs.mru_ghost_data_lsize 0
vfs.zfs.mfu_size 3925990912
vfs.zfs.mfu_metadata_lsize 3901947392
vfs.zfs.mfu_data_lsize 7000064
vfs.zfs.mfu_ghost_size 0
vfs.zfs.mfu_ghost_metadata_lsize 0
vfs.zfs.mfu_ghost_data_lsize 0
vfs.zfs.l2c_only_size 0
vfs.zfs.dedup.prefetch 1
vfs.zfs.nopwrite_enabled 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.max_recordsize 1048576
vfs.zfs.dirty_data_max 3429735628
vfs.zfs.dirty_data_max_max 4294967296
vfs.zfs.dirty_data_max_percent 10
vfs.zfs.dirty_data_sync 67108864
vfs.zfs.delay_min_dirty_percent 60
vfs.zfs.delay_scale 500000
vfs.zfs.prefetch_disable 0
vfs.zfs.zfetch.max_streams 8
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.top_maxinflight 32
vfs.zfs.resilver_delay 2
vfs.zfs.scrub_delay 4
vfs.zfs.scan_idle 50
vfs.zfs.scan_min_time_ms 1000
vfs.zfs.free_min_time_ms 1000
vfs.zfs.resilver_min_time_ms 3000
vfs.zfs.no_scrub_io 0
vfs.zfs.no_scrub_prefetch 0
vfs.zfs.free_max_blocks -1
vfs.zfs.metaslab.gang_bang 16777217
vfs.zfs.metaslab.fragmentation_threshold70
vfs.zfs.metaslab.debug_load 0
vfs.zfs.metaslab.debug_unload 0
vfs.zfs.metaslab.df_alloc_threshold 131072
vfs.zfs.metaslab.df_free_pct 4
vfs.zfs.metaslab.min_alloc_size 33554432
vfs.zfs.metaslab.load_pct 50
vfs.zfs.metaslab.unload_delay 8
vfs.zfs.metaslab.preload_limit 3
vfs.zfs.metaslab.preload_enabled 1
vfs.zfs.metaslab.fragmentation_factor_enabled1
vfs.zfs.metaslab.lba_weighting_enabled 1
vfs.zfs.metaslab.bias_enabled 1
vfs.zfs.condense_pct 200
vfs.zfs.mg_noalloc_threshold 0
vfs.zfs.mg_fragmentation_threshold 85
vfs.zfs.check_hostid 1
vfs.zfs.spa_load_verify_maxinflight 10000
vfs.zfs.spa_load_verify_metadata 1
vfs.zfs.spa_load_verify_data 1
vfs.zfs.recover 0
vfs.zfs.deadman_synctime_ms 1000000
vfs.zfs.deadman_checktime_ms 5000
vfs.zfs.deadman_enabled 1
vfs.zfs.spa_asize_inflation 24
vfs.zfs.spa_slop_shift 5
vfs.zfs.space_map_blksz 4096
vfs.zfs.txg.timeout 5
vfs.zfs.vdev.metaslabs_per_vdev 200
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.trim_on_init 1
vfs.zfs.vdev.mirror.rotating_inc 0
vfs.zfs.vdev.mirror.rotating_seek_inc 5
vfs.zfs.vdev.mirror.rotating_seek_offset1048576
vfs.zfs.vdev.mirror.non_rotating_inc 0
vfs.zfs.vdev.mirror.non_rotating_seek_inc1
vfs.zfs.vdev.async_write_active_min_dirty_percent30
vfs.zfs.vdev.async_write_active_max_dirty_percent60
vfs.zfs.vdev.max_active 1000
vfs.zfs.vdev.sync_read_min_active 10
vfs.zfs.vdev.sync_read_max_active 10
vfs.zfs.vdev.sync_write_min_active 10
vfs.zfs.vdev.sync_write_max_active 10
vfs.zfs.vdev.async_read_min_active 1
vfs.zfs.vdev.async_read_max_active 3
vfs.zfs.vdev.async_write_min_active 1
vfs.zfs.vdev.async_write_max_active 10
vfs.zfs.vdev.scrub_min_active 1
vfs.zfs.vdev.scrub_max_active 2
vfs.zfs.vdev.trim_min_active 1
vfs.zfs.vdev.trim_max_active 64
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.vdev.bio_delete_disable 0
vfs.zfs.vdev.trim_max_bytes 2147483648
vfs.zfs.vdev.trim_max_pending 64
vfs.zfs.max_auto_ashift 13
vfs.zfs.min_auto_ashift 9
vfs.zfs.zil_replay_disable 0
vfs.zfs.cache_flush_disable 0
vfs.zfs.zio.use_uma 1
vfs.zfs.zio.exclude_metadata 0
vfs.zfs.sync_pass_deferred_free 2
vfs.zfs.sync_pass_dont_compress 5
vfs.zfs.sync_pass_rewrite 2
vfs.zfs.snapshot_list_prefetch 0
vfs.zfs.super_owner 0
vfs.zfs.debug 0
vfs.zfs.version.ioctl 4
vfs.zfs.version.acl 1
vfs.zfs.version.spa 5000
vfs.zfs.version.zpl 5
vfs.zfs.vol.mode 1
vfs.zfs.vol.unmap_enabled 1
vfs.zfs.trim.enabled 1
vfs.zfs.trim.txg_delay 32
vfs.zfs.trim.timeout 30
vfs.zfs.trim.max_interval 1
------------------------------------------------------------------------
More information about the freebsd-fs
mailing list