High CPU Interrupt using ZFS
Kaya Saman
kayasaman at gmail.com
Sun Jun 19 19:38:10 UTC 2016
Hi,
I have a strange problem and I'm not sure if anyone has ever experienced
this to help give me some advice on how to tackle it.
Basically I run ZFS as root FS mirrored over two drives which are
directly connected to the SATA connectors on a SuperMicro Xeon E5 server
based MB.
Then I have an LSI HBA connected to the remaining disks with various
ZPOOLs. The main pool has ZIL and L2ARC enabled.
As the majority of data is A/V content I disabled Prefetch as instructed
in the FreeBSD tuning tips guide.
https://www.freebsd.org/doc/handbook/zfs.html
For some reason after a period of time the CPU interrupt will just got
sky high and the system will totally bog down. My home drive is running
off the "Main Pool" too and when this happens it becomes inaccessible.
The system runs FBSD 10.3: 10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264:
Fri Mar 25 02:10:02 UTC 2016
ZPOOL List output:
# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH
ALTROOT
ZPOOL_2 27.2T 26.3T 884G - 41% 96% 1.00x ONLINE -
ZPOOL_3 298G 248G 50.2G - 34% 83% 1.00x ONLINE -
ZPOOL_4 1.81T 1.75T 66.4G - 25% 96% 1.00x ONLINE -
ZPOOL_5 186G 171G 14.9G - 62% 92% 1.00x ONLINE -
workspaces 119G 77.7G 41.3G - 56% 65% 1.00x ONLINE -
zroot 111G 88.9G 22.1G - 70% 80% 1.00x ONLINE -
The system has a Xeon E5 with 24GB RAM and 16GB of Swap space.
I also run 5x jails on this box.
1x for network based monitoring (munin, zabbix etc)
1x DB jail which runs Postgresql and Mysql
+ some others; they are all run off the ZRoot
Boot Loader info:
zfs_load="YES"
kern.ipc.semmni=6000000
kern.ipc.semmns=6000000
kern.ipc.semmnu=256
net.isr.numthreads=4
net.isr.maxthreads=4
net.isr.bindthreads=1
vfs.zfs.l2arc_noprefetch=1
Other information:
# camcontrol devlist
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 8 lun 0 (pass0,da0)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 10 lun 0 (pass1,da1)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 11 lun 0 (pass2,da2)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 12 lun 0 (pass3,da3)
<LSI SAS2X36 0e12> at scbus0 target 13 lun 0 (pass4,ses0)
<ATA Corsair Force GS 5.20> at scbus0 target 14 lun 0 (pass5,da4)
<ATA Corsair Force GS 5.20> at scbus0 target 15 lun 0 (pass6,da5)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 17 lun 0 (pass7,da6)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 18 lun 0 (pass8,da7)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 19 lun 0 (pass9,da8)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 20 lun 0 (pass10,da9)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 21 lun 0 (pass11,da10)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 22 lun 0 (pass12,da11)
<ATA OCZ-VERTEX4 1.5> at scbus0 target 29 lun 0 (pass13,da12)
<ATA ST9320423AS SDM1> at scbus0 target 30 lun 0 (pass14,da13)
<ATA ST9200420AS A> at scbus0 target 31 lun 0 (pass15,da14)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 34 lun 0 (pass16,da15)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 35 lun 0 (pass17,da16)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 36 lun 0 (pass18,da17)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 37 lun 0 (pass19,da18)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 38 lun 0 (pass20,da19)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 40 lun 0 (pass21,da20)
<ATA WDC WD20NPVX-00E 1A01> at scbus0 target 41 lun 0 (pass22,da21)
<Corsair Force GS 5.41> at scbus2 target 0 lun 0 (pass23,ada0)
<Corsair Force GS 5.20> at scbus3 target 0 lun 0 (pass24,ada1)
<AHCI SGPIO Enclosure 1.00 0001> at scbus8 target 0 lun 0 (pass25,ses1)
Sysctl output for ZFS:
# sysctl -a |grep zfs
2 PART diskid/DISK-1350790500009986007Fp2 229319956992 512 i 2 o
10737435648 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 PART diskid/DISK-1350790500009986007Fp1 10737418240 512 i 1 o 17408 ty
freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 PART diskid/DISK-13507905000099860071p2 229319956992 512 i 2 o
10737435648 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 PART diskid/DISK-13507905000099860071p1 10735321088 512 i 1 o 2097152
ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
2 PART diskid/DISK-14067903000097960BD7p3 119445590016 512 i 3 o
8590065664 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
1 PART ada0p3 119445590016 512 i 3 o 8590065664 ty freebsd-zfs xs GPT xt
516e7cba-6ecf-11d6-8ff8-00022d09712b
z0xfffff80012422d00 [shape=box,label="ZFS::VDEV\nzfs::vdev\nr#4"];
<name>zfs::vdev</name>
<type>freebsd-zfs</type>
<type>freebsd-zfs</type>
<type>freebsd-zfs</type>
<type>freebsd-zfs</type>
<type>freebsd-zfs</type>
<type>freebsd-zfs</type>
vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 1
vfs.zfs.vol.unmap_enabled: 1
vfs.zfs.vol.mode: 1
vfs.zfs.version.zpl: 5
vfs.zfs.version.spa: 5000
vfs.zfs.version.acl: 1
vfs.zfs.version.ioctl: 5
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
vfs.zfs.sync_pass_rewrite: 2
vfs.zfs.sync_pass_dont_compress: 5
vfs.zfs.sync_pass_deferred_free: 2
vfs.zfs.zio.exclude_metadata: 0
vfs.zfs.zio.use_uma: 1
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_replay_disable: 0
vfs.zfs.min_auto_ashift: 9
vfs.zfs.max_auto_ashift: 13
vfs.zfs.vdev.trim_max_pending: 10000
vfs.zfs.vdev.bio_delete_disable: 0
vfs.zfs.vdev.bio_flush_disable: 0
vfs.zfs.vdev.write_gap_limit: 4096
vfs.zfs.vdev.read_gap_limit: 32768
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.scrub_max_active: 2
vfs.zfs.vdev.scrub_min_active: 1
vfs.zfs.vdev.async_write_max_active: 10
vfs.zfs.vdev.async_write_min_active: 1
vfs.zfs.vdev.async_read_max_active: 3
vfs.zfs.vdev.async_read_min_active: 1
vfs.zfs.vdev.sync_write_max_active: 10
vfs.zfs.vdev.sync_write_min_active: 10
vfs.zfs.vdev.sync_read_max_active: 10
vfs.zfs.vdev.sync_read_min_active: 10
vfs.zfs.vdev.max_active: 1000
vfs.zfs.vdev.async_write_active_max_dirty_percent: 60
vfs.zfs.vdev.async_write_active_min_dirty_percent: 30
vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1
vfs.zfs.vdev.mirror.non_rotating_inc: 0
vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576
vfs.zfs.vdev.mirror.rotating_seek_inc: 5
vfs.zfs.vdev.mirror.rotating_inc: 0
vfs.zfs.vdev.trim_on_init: 1
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 0
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.metaslabs_per_vdev: 200
vfs.zfs.txg.timeout: 5
vfs.zfs.space_map_blksz: 4096
vfs.zfs.spa_slop_shift: 5
vfs.zfs.spa_asize_inflation: 24
vfs.zfs.deadman_enabled: 1
vfs.zfs.deadman_checktime_ms: 5000
vfs.zfs.deadman_synctime_ms: 1000000
vfs.zfs.recover: 0
vfs.zfs.spa_load_verify_data: 1
vfs.zfs.spa_load_verify_metadata: 1
vfs.zfs.spa_load_verify_maxinflight: 10000
vfs.zfs.check_hostid: 1
vfs.zfs.mg_fragmentation_threshold: 85
vfs.zfs.mg_noalloc_threshold: 0
vfs.zfs.condense_pct: 200
vfs.zfs.metaslab.bias_enabled: 1
vfs.zfs.metaslab.lba_weighting_enabled: 1
vfs.zfs.metaslab.fragmentation_factor_enabled: 1
vfs.zfs.metaslab.preload_enabled: 1
vfs.zfs.metaslab.preload_limit: 3
vfs.zfs.metaslab.unload_delay: 8
vfs.zfs.metaslab.load_pct: 50
vfs.zfs.metaslab.min_alloc_size: 33554432
vfs.zfs.metaslab.df_free_pct: 4
vfs.zfs.metaslab.df_alloc_threshold: 131072
vfs.zfs.metaslab.debug_unload: 0
vfs.zfs.metaslab.debug_load: 0
vfs.zfs.metaslab.fragmentation_threshold: 70
vfs.zfs.metaslab.gang_bang: 16777217
vfs.zfs.free_bpobj_enabled: 1
vfs.zfs.free_max_blocks: 18446744073709551615
vfs.zfs.no_scrub_prefetch: 0
vfs.zfs.no_scrub_io: 0
vfs.zfs.resilver_min_time_ms: 3000
vfs.zfs.free_min_time_ms: 1000
vfs.zfs.scan_min_time_ms: 1000
vfs.zfs.scan_idle: 50
vfs.zfs.scrub_delay: 4
vfs.zfs.resilver_delay: 2
vfs.zfs.top_maxinflight: 32
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.max_distance: 8388608
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 0
vfs.zfs.delay_scale: 500000
vfs.zfs.delay_min_dirty_percent: 60
vfs.zfs.dirty_data_sync: 67108864
vfs.zfs.dirty_data_max_percent: 10
vfs.zfs.dirty_data_max_max: 4294967296
vfs.zfs.dirty_data_max: 2570453401
vfs.zfs.max_recordsize: 1048576
vfs.zfs.mdcomp_disable: 0
vfs.zfs.nopwrite_enabled: 1
vfs.zfs.dedup.prefetch: 1
vfs.zfs.l2c_only_size: 0
vfs.zfs.mfu_ghost_data_lsize: 3288968704
vfs.zfs.mfu_ghost_metadata_lsize: 5136092672
vfs.zfs.mfu_ghost_size: 8425061376
vfs.zfs.mfu_data_lsize: 8574981632
vfs.zfs.mfu_metadata_lsize: 68123648
vfs.zfs.mfu_size: 8745474560
vfs.zfs.mru_ghost_data_lsize: 5324684800
vfs.zfs.mru_ghost_metadata_lsize: 923847680
vfs.zfs.mru_ghost_size: 6248532480
vfs.zfs.mru_data_lsize: 1456756224
vfs.zfs.mru_metadata_lsize: 1278004224
vfs.zfs.mru_size: 2862586368
vfs.zfs.anon_data_lsize: 0
vfs.zfs.anon_metadata_lsize: 0
vfs.zfs.anon_size: 2841088
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 134217728
vfs.zfs.l2arc_write_max: 67108864
vfs.zfs.arc_meta_limit: 5979973632
vfs.zfs.arc_free_target: 42350
vfs.zfs.arc_shrink_shift: 7
vfs.zfs.arc_average_blocksize: 8192
vfs.zfs.arc_min: 2989986816
vfs.zfs.arc_max: 23919894528
debug.zfs_flags: 0
kstat.zfs.misc.vdev_cache_stats.misses: 0
kstat.zfs.misc.vdev_cache_stats.hits: 0
kstat.zfs.misc.vdev_cache_stats.delegations: 0
kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 3034069
kstat.zfs.misc.arcstats.sync_wait_for_async: 1779944
kstat.zfs.misc.arcstats.arc_meta_min: 1494993408
kstat.zfs.misc.arcstats.arc_meta_max: 12233249160
kstat.zfs.misc.arcstats.arc_meta_limit: 5979973632
kstat.zfs.misc.arcstats.arc_meta_used: 4638138472
kstat.zfs.misc.arcstats.duplicate_reads: 1709068
kstat.zfs.misc.arcstats.duplicate_buffers_size: 0
kstat.zfs.misc.arcstats.duplicate_buffers: 0
kstat.zfs.misc.arcstats.memory_throttle_count: 0
kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 2200
kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 4772510
kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 561679501832704
kstat.zfs.misc.arcstats.l2_write_pios: 377935
kstat.zfs.misc.arcstats.l2_write_buffer_iter: 1193136
kstat.zfs.misc.arcstats.l2_write_full: 148
kstat.zfs.misc.arcstats.l2_write_not_cacheable: 264598116
kstat.zfs.misc.arcstats.l2_write_io_in_progress: 83
kstat.zfs.misc.arcstats.l2_write_in_l2: 5284665382
kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 14626890890
kstat.zfs.misc.arcstats.l2_write_passed_headroom: 3318575
kstat.zfs.misc.arcstats.l2_write_trylock_fail: 6328431
kstat.zfs.misc.arcstats.l2_compress_failures: 655251
kstat.zfs.misc.arcstats.l2_compress_zeros: 0
kstat.zfs.misc.arcstats.l2_compress_successes: 1205377
kstat.zfs.misc.arcstats.l2_hdr_size: 63556704
kstat.zfs.misc.arcstats.l2_asize: 84595239936
kstat.zfs.misc.arcstats.l2_size: 93178570752
kstat.zfs.misc.arcstats.l2_io_error: 0
kstat.zfs.misc.arcstats.l2_cksum_bad: 0
kstat.zfs.misc.arcstats.l2_abort_lowmem: 42
kstat.zfs.misc.arcstats.l2_cdata_free_on_write: 41
kstat.zfs.misc.arcstats.l2_free_on_write: 722
kstat.zfs.misc.arcstats.l2_evict_l1cached: 0
kstat.zfs.misc.arcstats.l2_evict_reading: 0
kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
kstat.zfs.misc.arcstats.l2_writes_lock_retry: 63
kstat.zfs.misc.arcstats.l2_writes_error: 0
kstat.zfs.misc.arcstats.l2_writes_done: 377935
kstat.zfs.misc.arcstats.l2_writes_sent: 377935
kstat.zfs.misc.arcstats.l2_write_bytes: 101118255104
kstat.zfs.misc.arcstats.l2_read_bytes: 59571878912
kstat.zfs.misc.arcstats.l2_rw_clash: 0
kstat.zfs.misc.arcstats.l2_feeds: 1193136
kstat.zfs.misc.arcstats.l2_misses: 137818470
kstat.zfs.misc.arcstats.l2_hits: 3613135
kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 5136092672
kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 3722561024
kstat.zfs.misc.arcstats.mfu_ghost_size: 8858653696
kstat.zfs.misc.arcstats.mfu_evictable_metadata: 68123648
kstat.zfs.misc.arcstats.mfu_evictable_data: 8575112704
kstat.zfs.misc.arcstats.mfu_size: 8745605632
kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 923847680
kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 5324684800
kstat.zfs.misc.arcstats.mru_ghost_size: 6248532480
kstat.zfs.misc.arcstats.mru_evictable_metadata: 1278004224
kstat.zfs.misc.arcstats.mru_evictable_data: 1457411584
kstat.zfs.misc.arcstats.mru_size: 2863241728
kstat.zfs.misc.arcstats.anon_evictable_metadata: 0
kstat.zfs.misc.arcstats.anon_evictable_data: 0
kstat.zfs.misc.arcstats.anon_size: 2038272
kstat.zfs.misc.arcstats.other_size: 2797801256
kstat.zfs.misc.arcstats.metadata_size: 1576374272
kstat.zfs.misc.arcstats.data_size: 10034527744
kstat.zfs.misc.arcstats.hdr_size: 200406240
kstat.zfs.misc.arcstats.size: 14672666216
kstat.zfs.misc.arcstats.c_max: 23919894528
kstat.zfs.misc.arcstats.c_min: 2989986816
kstat.zfs.misc.arcstats.c: 14673666683
kstat.zfs.misc.arcstats.p: 8668447917
kstat.zfs.misc.arcstats.hash_chain_max: 7
kstat.zfs.misc.arcstats.hash_chains: 219061
kstat.zfs.misc.arcstats.hash_collisions: 33107789
kstat.zfs.misc.arcstats.hash_elements_max: 1529284
kstat.zfs.misc.arcstats.hash_elements: 1529163
kstat.zfs.misc.arcstats.evict_l2_skip: 0
kstat.zfs.misc.arcstats.evict_l2_ineligible: 353901531136
kstat.zfs.misc.arcstats.evict_l2_eligible: 611148992512
kstat.zfs.misc.arcstats.evict_l2_cached: 471776311808
kstat.zfs.misc.arcstats.evict_not_enough: 2164
kstat.zfs.misc.arcstats.evict_skip: 232562
kstat.zfs.misc.arcstats.mutex_miss: 17547
kstat.zfs.misc.arcstats.deleted: 10350064
kstat.zfs.misc.arcstats.allocated: 172235521
kstat.zfs.misc.arcstats.mfu_ghost_hits: 8494679
kstat.zfs.misc.arcstats.mfu_hits: 1457647309
kstat.zfs.misc.arcstats.mru_ghost_hits: 5765227
kstat.zfs.misc.arcstats.mru_hits: 90829356
kstat.zfs.misc.arcstats.prefetch_metadata_misses: 4657105
kstat.zfs.misc.arcstats.prefetch_metadata_hits: 14515029
kstat.zfs.misc.arcstats.prefetch_data_misses: 6610395
kstat.zfs.misc.arcstats.prefetch_data_hits: 6837739
kstat.zfs.misc.arcstats.demand_metadata_misses: 127929204
kstat.zfs.misc.arcstats.demand_metadata_hits: 404655615
kstat.zfs.misc.arcstats.demand_data_misses: 2235496
kstat.zfs.misc.arcstats.demand_data_hits: 1138662563
kstat.zfs.misc.arcstats.misses: 141432200
kstat.zfs.misc.arcstats.hits: 1564670946
kstat.zfs.misc.zcompstats.skipped_insufficient_gain: 4581339
kstat.zfs.misc.zcompstats.empty: 842987
kstat.zfs.misc.zcompstats.attempts: 121463608
kstat.zfs.misc.zfetchstats.max_streams: 2029717049
kstat.zfs.misc.zfetchstats.misses: 2043239863
kstat.zfs.misc.zfetchstats.hits: 15544425
kstat.zfs.misc.xuio_stats.write_buf_nocopy: 1453761
kstat.zfs.misc.xuio_stats.write_buf_copied: 0
kstat.zfs.misc.xuio_stats.read_buf_nocopy: 0
kstat.zfs.misc.xuio_stats.read_buf_copied: 0
kstat.zfs.misc.xuio_stats.onloan_write_buf: 0
kstat.zfs.misc.xuio_stats.onloan_read_buf: 0
kstat.zfs.misc.zio_trim.failed: 0
kstat.zfs.misc.zio_trim.unsupported: 867
kstat.zfs.misc.zio_trim.success: 165333478
kstat.zfs.misc.zio_trim.bytes: 9734294003712
security.jail.param.allow.mount.zfs: 0
security.jail.mount_zfs_allowed: 0
I really don't know but could it be a conflict between the MB SATA ports
and LSI HBA?? As upon startup there do seem to be some ATA Error
messages in dmesg...
So more of a physical HW issue then FS based?
Or is it due to the term "bursty IO" that happens with ZFS... either way
I have been looking at this for months trying to figure things out but
other then a reboot nothing I do makes things better! Turning off my
monitoring jail does help on occasion but outside of that I'm lost.
I have another NAS based system with UFS root on SSD also yet ZPOOLs
over the various large mechanical drives but never run into this
particular issue!
Would anyone be able to help???
Many thanks.
Kaya
More information about the freebsd-fs
mailing list