High CPU Interrupt using ZFS
Steven Hartland
killing at multiplay.co.uk
Sun Jun 19 21:47:22 UTC 2016
You usage levels are really high I would recommend keeping things below
80% otherwise when new data is written its much more costly to locate
free space.
On 19/06/2016 20:38, Kaya Saman wrote:
> Hi,
>
>
> I have a strange problem and I'm not sure if anyone has ever
> experienced this to help give me some advice on how to tackle it.
>
>
> Basically I run ZFS as root FS mirrored over two drives which are
> directly connected to the SATA connectors on a SuperMicro Xeon E5
> server based MB.
>
>
> Then I have an LSI HBA connected to the remaining disks with various
> ZPOOLs. The main pool has ZIL and L2ARC enabled.
>
>
> As the majority of data is A/V content I disabled Prefetch as
> instructed in the FreeBSD tuning tips guide.
>
>
> https://www.freebsd.org/doc/handbook/zfs.html
>
>
> For some reason after a period of time the CPU interrupt will just got
> sky high and the system will totally bog down. My home drive is
> running off the "Main Pool" too and when this happens it becomes
> inaccessible.
>
>
> The system runs FBSD 10.3: 10.3-RELEASE FreeBSD 10.3-RELEASE #0
> r297264: Fri Mar 25 02:10:02 UTC 2016
>
>
> ZPOOL List output:
>
>
> # zpool list
> NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH
> ALTROOT
> ZPOOL_2 27.2T 26.3T 884G - 41% 96% 1.00x ONLINE -
> ZPOOL_3 298G 248G 50.2G - 34% 83% 1.00x ONLINE -
> ZPOOL_4 1.81T 1.75T 66.4G - 25% 96% 1.00x ONLINE -
> ZPOOL_5 186G 171G 14.9G - 62% 92% 1.00x ONLINE -
> workspaces 119G 77.7G 41.3G - 56% 65% 1.00x ONLINE -
> zroot 111G 88.9G 22.1G - 70% 80% 1.00x ONLINE -
>
>
> The system has a Xeon E5 with 24GB RAM and 16GB of Swap space.
>
>
> I also run 5x jails on this box.
>
> 1x for network based monitoring (munin, zabbix etc)
>
> 1x DB jail which runs Postgresql and Mysql
>
>
> + some others; they are all run off the ZRoot
>
>
> Boot Loader info:
>
>
> zfs_load="YES"
>
> kern.ipc.semmni=6000000
> kern.ipc.semmns=6000000
> kern.ipc.semmnu=256
>
> net.isr.numthreads=4
> net.isr.maxthreads=4
> net.isr.bindthreads=1
>
> vfs.zfs.l2arc_noprefetch=1
>
>
> Other information:
>
>
> # camcontrol devlist
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 8 lun 0 (pass0,da0)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 10 lun 0 (pass1,da1)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 11 lun 0 (pass2,da2)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 12 lun 0 (pass3,da3)
> <LSI SAS2X36 0e12> at scbus0 target 13 lun 0 (pass4,ses0)
> <ATA Corsair Force GS 5.20> at scbus0 target 14 lun 0 (pass5,da4)
> <ATA Corsair Force GS 5.20> at scbus0 target 15 lun 0 (pass6,da5)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 17 lun 0 (pass7,da6)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 18 lun 0 (pass8,da7)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 19 lun 0 (pass9,da8)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 20 lun 0 (pass10,da9)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 21 lun 0
> (pass11,da10)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 22 lun 0
> (pass12,da11)
> <ATA OCZ-VERTEX4 1.5> at scbus0 target 29 lun 0
> (pass13,da12)
> <ATA ST9320423AS SDM1> at scbus0 target 30 lun 0
> (pass14,da13)
> <ATA ST9200420AS A> at scbus0 target 31 lun 0
> (pass15,da14)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 34 lun 0
> (pass16,da15)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 35 lun 0
> (pass17,da16)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 36 lun 0
> (pass18,da17)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 37 lun 0
> (pass19,da18)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 38 lun 0
> (pass20,da19)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 40 lun 0
> (pass21,da20)
> <ATA WDC WD20NPVX-00E 1A01> at scbus0 target 41 lun 0
> (pass22,da21)
> <Corsair Force GS 5.41> at scbus2 target 0 lun 0 (pass23,ada0)
> <Corsair Force GS 5.20> at scbus3 target 0 lun 0 (pass24,ada1)
> <AHCI SGPIO Enclosure 1.00 0001> at scbus8 target 0 lun 0 (pass25,ses1)
>
>
> Sysctl output for ZFS:
>
>
> # sysctl -a |grep zfs
> 2 PART diskid/DISK-1350790500009986007Fp2 229319956992 512 i 2 o
> 10737435648 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
> 2 PART diskid/DISK-1350790500009986007Fp1 10737418240 512 i 1 o 17408
> ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
> 2 PART diskid/DISK-13507905000099860071p2 229319956992 512 i 2 o
> 10737435648 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
> 2 PART diskid/DISK-13507905000099860071p1 10735321088 512 i 1 o
> 2097152 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
> 2 PART diskid/DISK-14067903000097960BD7p3 119445590016 512 i 3 o
> 8590065664 ty freebsd-zfs xs GPT xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
> 1 PART ada0p3 119445590016 512 i 3 o 8590065664 ty freebsd-zfs xs GPT
> xt 516e7cba-6ecf-11d6-8ff8-00022d09712b
> z0xfffff80012422d00 [shape=box,label="ZFS::VDEV\nzfs::vdev\nr#4"];
> <name>zfs::vdev</name>
> <type>freebsd-zfs</type>
> <type>freebsd-zfs</type>
> <type>freebsd-zfs</type>
> <type>freebsd-zfs</type>
> <type>freebsd-zfs</type>
> <type>freebsd-zfs</type>
> vfs.zfs.trim.max_interval: 1
> vfs.zfs.trim.timeout: 30
> vfs.zfs.trim.txg_delay: 32
> vfs.zfs.trim.enabled: 1
> vfs.zfs.vol.unmap_enabled: 1
> vfs.zfs.vol.mode: 1
> vfs.zfs.version.zpl: 5
> vfs.zfs.version.spa: 5000
> vfs.zfs.version.acl: 1
> vfs.zfs.version.ioctl: 5
> vfs.zfs.debug: 0
> vfs.zfs.super_owner: 0
> vfs.zfs.sync_pass_rewrite: 2
> vfs.zfs.sync_pass_dont_compress: 5
> vfs.zfs.sync_pass_deferred_free: 2
> vfs.zfs.zio.exclude_metadata: 0
> vfs.zfs.zio.use_uma: 1
> vfs.zfs.cache_flush_disable: 0
> vfs.zfs.zil_replay_disable: 0
> vfs.zfs.min_auto_ashift: 9
> vfs.zfs.max_auto_ashift: 13
> vfs.zfs.vdev.trim_max_pending: 10000
> vfs.zfs.vdev.bio_delete_disable: 0
> vfs.zfs.vdev.bio_flush_disable: 0
> vfs.zfs.vdev.write_gap_limit: 4096
> vfs.zfs.vdev.read_gap_limit: 32768
> vfs.zfs.vdev.aggregation_limit: 131072
> vfs.zfs.vdev.trim_max_active: 64
> vfs.zfs.vdev.trim_min_active: 1
> vfs.zfs.vdev.scrub_max_active: 2
> vfs.zfs.vdev.scrub_min_active: 1
> vfs.zfs.vdev.async_write_max_active: 10
> vfs.zfs.vdev.async_write_min_active: 1
> vfs.zfs.vdev.async_read_max_active: 3
> vfs.zfs.vdev.async_read_min_active: 1
> vfs.zfs.vdev.sync_write_max_active: 10
> vfs.zfs.vdev.sync_write_min_active: 10
> vfs.zfs.vdev.sync_read_max_active: 10
> vfs.zfs.vdev.sync_read_min_active: 10
> vfs.zfs.vdev.max_active: 1000
> vfs.zfs.vdev.async_write_active_max_dirty_percent: 60
> vfs.zfs.vdev.async_write_active_min_dirty_percent: 30
> vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1
> vfs.zfs.vdev.mirror.non_rotating_inc: 0
> vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576
> vfs.zfs.vdev.mirror.rotating_seek_inc: 5
> vfs.zfs.vdev.mirror.rotating_inc: 0
> vfs.zfs.vdev.trim_on_init: 1
> vfs.zfs.vdev.cache.bshift: 16
> vfs.zfs.vdev.cache.size: 0
> vfs.zfs.vdev.cache.max: 16384
> vfs.zfs.vdev.metaslabs_per_vdev: 200
> vfs.zfs.txg.timeout: 5
> vfs.zfs.space_map_blksz: 4096
> vfs.zfs.spa_slop_shift: 5
> vfs.zfs.spa_asize_inflation: 24
> vfs.zfs.deadman_enabled: 1
> vfs.zfs.deadman_checktime_ms: 5000
> vfs.zfs.deadman_synctime_ms: 1000000
> vfs.zfs.recover: 0
> vfs.zfs.spa_load_verify_data: 1
> vfs.zfs.spa_load_verify_metadata: 1
> vfs.zfs.spa_load_verify_maxinflight: 10000
> vfs.zfs.check_hostid: 1
> vfs.zfs.mg_fragmentation_threshold: 85
> vfs.zfs.mg_noalloc_threshold: 0
> vfs.zfs.condense_pct: 200
> vfs.zfs.metaslab.bias_enabled: 1
> vfs.zfs.metaslab.lba_weighting_enabled: 1
> vfs.zfs.metaslab.fragmentation_factor_enabled: 1
> vfs.zfs.metaslab.preload_enabled: 1
> vfs.zfs.metaslab.preload_limit: 3
> vfs.zfs.metaslab.unload_delay: 8
> vfs.zfs.metaslab.load_pct: 50
> vfs.zfs.metaslab.min_alloc_size: 33554432
> vfs.zfs.metaslab.df_free_pct: 4
> vfs.zfs.metaslab.df_alloc_threshold: 131072
> vfs.zfs.metaslab.debug_unload: 0
> vfs.zfs.metaslab.debug_load: 0
> vfs.zfs.metaslab.fragmentation_threshold: 70
> vfs.zfs.metaslab.gang_bang: 16777217
> vfs.zfs.free_bpobj_enabled: 1
> vfs.zfs.free_max_blocks: 18446744073709551615
> vfs.zfs.no_scrub_prefetch: 0
> vfs.zfs.no_scrub_io: 0
> vfs.zfs.resilver_min_time_ms: 3000
> vfs.zfs.free_min_time_ms: 1000
> vfs.zfs.scan_min_time_ms: 1000
> vfs.zfs.scan_idle: 50
> vfs.zfs.scrub_delay: 4
> vfs.zfs.resilver_delay: 2
> vfs.zfs.top_maxinflight: 32
> vfs.zfs.zfetch.array_rd_sz: 1048576
> vfs.zfs.zfetch.max_distance: 8388608
> vfs.zfs.zfetch.min_sec_reap: 2
> vfs.zfs.zfetch.max_streams: 8
> vfs.zfs.prefetch_disable: 0
> vfs.zfs.delay_scale: 500000
> vfs.zfs.delay_min_dirty_percent: 60
> vfs.zfs.dirty_data_sync: 67108864
> vfs.zfs.dirty_data_max_percent: 10
> vfs.zfs.dirty_data_max_max: 4294967296
> vfs.zfs.dirty_data_max: 2570453401
> vfs.zfs.max_recordsize: 1048576
> vfs.zfs.mdcomp_disable: 0
> vfs.zfs.nopwrite_enabled: 1
> vfs.zfs.dedup.prefetch: 1
> vfs.zfs.l2c_only_size: 0
> vfs.zfs.mfu_ghost_data_lsize: 3288968704
> vfs.zfs.mfu_ghost_metadata_lsize: 5136092672
> vfs.zfs.mfu_ghost_size: 8425061376
> vfs.zfs.mfu_data_lsize: 8574981632
> vfs.zfs.mfu_metadata_lsize: 68123648
> vfs.zfs.mfu_size: 8745474560
> vfs.zfs.mru_ghost_data_lsize: 5324684800
> vfs.zfs.mru_ghost_metadata_lsize: 923847680
> vfs.zfs.mru_ghost_size: 6248532480
> vfs.zfs.mru_data_lsize: 1456756224
> vfs.zfs.mru_metadata_lsize: 1278004224
> vfs.zfs.mru_size: 2862586368
> vfs.zfs.anon_data_lsize: 0
> vfs.zfs.anon_metadata_lsize: 0
> vfs.zfs.anon_size: 2841088
> vfs.zfs.l2arc_norw: 1
> vfs.zfs.l2arc_feed_again: 1
> vfs.zfs.l2arc_noprefetch: 1
> vfs.zfs.l2arc_feed_min_ms: 200
> vfs.zfs.l2arc_feed_secs: 1
> vfs.zfs.l2arc_headroom: 2
> vfs.zfs.l2arc_write_boost: 134217728
> vfs.zfs.l2arc_write_max: 67108864
> vfs.zfs.arc_meta_limit: 5979973632
> vfs.zfs.arc_free_target: 42350
> vfs.zfs.arc_shrink_shift: 7
> vfs.zfs.arc_average_blocksize: 8192
> vfs.zfs.arc_min: 2989986816
> vfs.zfs.arc_max: 23919894528
> debug.zfs_flags: 0
> kstat.zfs.misc.vdev_cache_stats.misses: 0
> kstat.zfs.misc.vdev_cache_stats.hits: 0
> kstat.zfs.misc.vdev_cache_stats.delegations: 0
> kstat.zfs.misc.arcstats.demand_hit_predictive_prefetch: 3034069
> kstat.zfs.misc.arcstats.sync_wait_for_async: 1779944
> kstat.zfs.misc.arcstats.arc_meta_min: 1494993408
> kstat.zfs.misc.arcstats.arc_meta_max: 12233249160
> kstat.zfs.misc.arcstats.arc_meta_limit: 5979973632
> kstat.zfs.misc.arcstats.arc_meta_used: 4638138472
> kstat.zfs.misc.arcstats.duplicate_reads: 1709068
> kstat.zfs.misc.arcstats.duplicate_buffers_size: 0
> kstat.zfs.misc.arcstats.duplicate_buffers: 0
> kstat.zfs.misc.arcstats.memory_throttle_count: 0
> kstat.zfs.misc.arcstats.l2_write_buffer_list_null_iter: 2200
> kstat.zfs.misc.arcstats.l2_write_buffer_list_iter: 4772510
> kstat.zfs.misc.arcstats.l2_write_buffer_bytes_scanned: 561679501832704
> kstat.zfs.misc.arcstats.l2_write_pios: 377935
> kstat.zfs.misc.arcstats.l2_write_buffer_iter: 1193136
> kstat.zfs.misc.arcstats.l2_write_full: 148
> kstat.zfs.misc.arcstats.l2_write_not_cacheable: 264598116
> kstat.zfs.misc.arcstats.l2_write_io_in_progress: 83
> kstat.zfs.misc.arcstats.l2_write_in_l2: 5284665382
> kstat.zfs.misc.arcstats.l2_write_spa_mismatch: 14626890890
> kstat.zfs.misc.arcstats.l2_write_passed_headroom: 3318575
> kstat.zfs.misc.arcstats.l2_write_trylock_fail: 6328431
> kstat.zfs.misc.arcstats.l2_compress_failures: 655251
> kstat.zfs.misc.arcstats.l2_compress_zeros: 0
> kstat.zfs.misc.arcstats.l2_compress_successes: 1205377
> kstat.zfs.misc.arcstats.l2_hdr_size: 63556704
> kstat.zfs.misc.arcstats.l2_asize: 84595239936
> kstat.zfs.misc.arcstats.l2_size: 93178570752
> kstat.zfs.misc.arcstats.l2_io_error: 0
> kstat.zfs.misc.arcstats.l2_cksum_bad: 0
> kstat.zfs.misc.arcstats.l2_abort_lowmem: 42
> kstat.zfs.misc.arcstats.l2_cdata_free_on_write: 41
> kstat.zfs.misc.arcstats.l2_free_on_write: 722
> kstat.zfs.misc.arcstats.l2_evict_l1cached: 0
> kstat.zfs.misc.arcstats.l2_evict_reading: 0
> kstat.zfs.misc.arcstats.l2_evict_lock_retry: 0
> kstat.zfs.misc.arcstats.l2_writes_lock_retry: 63
> kstat.zfs.misc.arcstats.l2_writes_error: 0
> kstat.zfs.misc.arcstats.l2_writes_done: 377935
> kstat.zfs.misc.arcstats.l2_writes_sent: 377935
> kstat.zfs.misc.arcstats.l2_write_bytes: 101118255104
> kstat.zfs.misc.arcstats.l2_read_bytes: 59571878912
> kstat.zfs.misc.arcstats.l2_rw_clash: 0
> kstat.zfs.misc.arcstats.l2_feeds: 1193136
> kstat.zfs.misc.arcstats.l2_misses: 137818470
> kstat.zfs.misc.arcstats.l2_hits: 3613135
> kstat.zfs.misc.arcstats.mfu_ghost_evictable_metadata: 5136092672
> kstat.zfs.misc.arcstats.mfu_ghost_evictable_data: 3722561024
> kstat.zfs.misc.arcstats.mfu_ghost_size: 8858653696
> kstat.zfs.misc.arcstats.mfu_evictable_metadata: 68123648
> kstat.zfs.misc.arcstats.mfu_evictable_data: 8575112704
> kstat.zfs.misc.arcstats.mfu_size: 8745605632
> kstat.zfs.misc.arcstats.mru_ghost_evictable_metadata: 923847680
> kstat.zfs.misc.arcstats.mru_ghost_evictable_data: 5324684800
> kstat.zfs.misc.arcstats.mru_ghost_size: 6248532480
> kstat.zfs.misc.arcstats.mru_evictable_metadata: 1278004224
> kstat.zfs.misc.arcstats.mru_evictable_data: 1457411584
> kstat.zfs.misc.arcstats.mru_size: 2863241728
> kstat.zfs.misc.arcstats.anon_evictable_metadata: 0
> kstat.zfs.misc.arcstats.anon_evictable_data: 0
> kstat.zfs.misc.arcstats.anon_size: 2038272
> kstat.zfs.misc.arcstats.other_size: 2797801256
> kstat.zfs.misc.arcstats.metadata_size: 1576374272
> kstat.zfs.misc.arcstats.data_size: 10034527744
> kstat.zfs.misc.arcstats.hdr_size: 200406240
> kstat.zfs.misc.arcstats.size: 14672666216
> kstat.zfs.misc.arcstats.c_max: 23919894528
> kstat.zfs.misc.arcstats.c_min: 2989986816
> kstat.zfs.misc.arcstats.c: 14673666683
> kstat.zfs.misc.arcstats.p: 8668447917
> kstat.zfs.misc.arcstats.hash_chain_max: 7
> kstat.zfs.misc.arcstats.hash_chains: 219061
> kstat.zfs.misc.arcstats.hash_collisions: 33107789
> kstat.zfs.misc.arcstats.hash_elements_max: 1529284
> kstat.zfs.misc.arcstats.hash_elements: 1529163
> kstat.zfs.misc.arcstats.evict_l2_skip: 0
> kstat.zfs.misc.arcstats.evict_l2_ineligible: 353901531136
> kstat.zfs.misc.arcstats.evict_l2_eligible: 611148992512
> kstat.zfs.misc.arcstats.evict_l2_cached: 471776311808
> kstat.zfs.misc.arcstats.evict_not_enough: 2164
> kstat.zfs.misc.arcstats.evict_skip: 232562
> kstat.zfs.misc.arcstats.mutex_miss: 17547
> kstat.zfs.misc.arcstats.deleted: 10350064
> kstat.zfs.misc.arcstats.allocated: 172235521
> kstat.zfs.misc.arcstats.mfu_ghost_hits: 8494679
> kstat.zfs.misc.arcstats.mfu_hits: 1457647309
> kstat.zfs.misc.arcstats.mru_ghost_hits: 5765227
> kstat.zfs.misc.arcstats.mru_hits: 90829356
> kstat.zfs.misc.arcstats.prefetch_metadata_misses: 4657105
> kstat.zfs.misc.arcstats.prefetch_metadata_hits: 14515029
> kstat.zfs.misc.arcstats.prefetch_data_misses: 6610395
> kstat.zfs.misc.arcstats.prefetch_data_hits: 6837739
> kstat.zfs.misc.arcstats.demand_metadata_misses: 127929204
> kstat.zfs.misc.arcstats.demand_metadata_hits: 404655615
> kstat.zfs.misc.arcstats.demand_data_misses: 2235496
> kstat.zfs.misc.arcstats.demand_data_hits: 1138662563
> kstat.zfs.misc.arcstats.misses: 141432200
> kstat.zfs.misc.arcstats.hits: 1564670946
> kstat.zfs.misc.zcompstats.skipped_insufficient_gain: 4581339
> kstat.zfs.misc.zcompstats.empty: 842987
> kstat.zfs.misc.zcompstats.attempts: 121463608
> kstat.zfs.misc.zfetchstats.max_streams: 2029717049
> kstat.zfs.misc.zfetchstats.misses: 2043239863
> kstat.zfs.misc.zfetchstats.hits: 15544425
> kstat.zfs.misc.xuio_stats.write_buf_nocopy: 1453761
> kstat.zfs.misc.xuio_stats.write_buf_copied: 0
> kstat.zfs.misc.xuio_stats.read_buf_nocopy: 0
> kstat.zfs.misc.xuio_stats.read_buf_copied: 0
> kstat.zfs.misc.xuio_stats.onloan_write_buf: 0
> kstat.zfs.misc.xuio_stats.onloan_read_buf: 0
> kstat.zfs.misc.zio_trim.failed: 0
> kstat.zfs.misc.zio_trim.unsupported: 867
> kstat.zfs.misc.zio_trim.success: 165333478
> kstat.zfs.misc.zio_trim.bytes: 9734294003712
> security.jail.param.allow.mount.zfs: 0
> security.jail.mount_zfs_allowed: 0
>
>
> I really don't know but could it be a conflict between the MB SATA
> ports and LSI HBA?? As upon startup there do seem to be some ATA Error
> messages in dmesg...
>
> So more of a physical HW issue then FS based?
>
> Or is it due to the term "bursty IO" that happens with ZFS... either
> way I have been looking at this for months trying to figure things out
> but other then a reboot nothing I do makes things better! Turning off
> my monitoring jail does help on occasion but outside of that I'm lost.
>
>
> I have another NAS based system with UFS root on SSD also yet ZPOOLs
> over the various large mechanical drives but never run into this
> particular issue!
>
>
> Would anyone be able to help???
>
>
> Many thanks.
>
>
> Kaya
>
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
More information about the freebsd-fs
mailing list