FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs

Tue Jun 20 20:26:12 UTC 2017

> -----Original Message-----
> From: Steven Hartland [mailto:killing at multiplay.co.uk]
> Sent: Tuesday, June 20, 2017 1:53 PM
> To: Caza, Aaron; freebsd-fs at freebsd.org
> Subject: Re: FreeBSD 11.1 Beta 2 ZFS performance degradation on SSDs
>
>
>
> On 20/06/2017 17:58, Caza, Aaron wrote:
> >
> >> Can you show the output from gstat -pd during this DD please.
> > dT: 1.001s  w: 1.000s
> >   L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
> >      0   4318   4318  34865    0.0      0      0    0.0      0      0    0.0   14.2| ada0
> >      0   4402   4402  35213    0.0      0      0    0.0      0      0    0.0   14.4| ada1
> >
> > dT: 1.002s  w: 1.000s
> >   L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
> >      1   4249   4249  34136    0.0      0      0    0.0      0      0    0.0   14.1| ada0
> >      0   4393   4393  35287    0.0      0      0    0.0      0      0    0.0   14.5| ada1
> You %busy is very low, so sounds like the bottleneck isn't in raw disk performance but somewhere else.
>
> Might be interesting to see if anything stands out in top -Sz and then press h for threads.
>

I rebooted the system to disable Trim so currently not degraded.

> You also mention ada, could you share:
> sysctl kern.cam
>

kern.cam.enc.emulate_array_devices: 1
kern.cam.sa.allow_io_split: 0
kern.cam.da.default_softtimeout: 0
kern.cam.da.send_ordered: 1
kern.cam.da.default_timeout: 60
kern.cam.da.retry_count: 4
kern.cam.da.poll_period: 3
kern.cam.ada.1.sort_io_queue: 0
kern.cam.ada.1.max_seq_zones: 0
kern.cam.ada.1.optimal_nonseq_zones: 0
kern.cam.ada.1.optimal_seq_zones: 0
kern.cam.ada.1.zone_support: None
kern.cam.ada.1.zone_mode: Not Zoned
kern.cam.ada.1.rotating: 0
kern.cam.ada.1.unmapped_io: 1
kern.cam.ada.1.write_cache: -1
kern.cam.ada.1.read_ahead: -1
kern.cam.ada.1.delete_method: DSM_TRIM
kern.cam.ada.0.sort_io_queue: 0
kern.cam.ada.0.max_seq_zones: 0
kern.cam.ada.0.optimal_nonseq_zones: 0
kern.cam.ada.0.optimal_seq_zones: 0
kern.cam.ada.0.zone_support: None
kern.cam.ada.0.zone_mode: Not Zoned
kern.cam.ada.0.rotating: 0
kern.cam.ada.0.unmapped_io: 1
kern.cam.ada.0.write_cache: -1
kern.cam.ada.0.read_ahead: -1
kern.cam.ada.0.delete_method: DSM_TRIM
kern.cam.ada.write_cache: 1
kern.cam.ada.read_ahead: 1
kern.cam.ada.spindown_suspend: 1
kern.cam.ada.spindown_shutdown: 1
kern.cam.ada.send_ordered: 1
kern.cam.ada.default_timeout: 30
kern.cam.ada.retry_count: 4
kern.cam.cd.timeout: 30000
kern.cam.cd.retry_count: 4
kern.cam.cd.poll_period: 3
kern.cam.scsi_delay: 5000
kern.cam.cam_srch_hi: 0
kern.cam.pmp.hide_special: 1
kern.cam.pmp.default_timeout: 30
kern.cam.pmp.retry_count: 1
kern.cam.debug_delay: 0
kern.cam.dflags: 0
kern.cam.num_doneqs: 2
kern.cam.xpt_generation: 13
kern.cam.boot_delay: 0
kern.cam.sort_io_queues: 1

> And:
> sysctl vfs.zfs
>

vfs.zfs.trim.max_interval: 1
vfs.zfs.trim.timeout: 30
vfs.zfs.trim.txg_delay: 32
vfs.zfs.trim.enabled: 0
vfs.zfs.vol.immediate_write_sz: 32768
vfs.zfs.vol.unmap_enabled: 1
vfs.zfs.vol.recursive: 0
vfs.zfs.vol.mode: 1
vfs.zfs.version.zpl: 5
vfs.zfs.version.spa: 5000
vfs.zfs.version.acl: 1
vfs.zfs.version.ioctl: 7
vfs.zfs.debug: 0
vfs.zfs.super_owner: 0
vfs.zfs.immediate_write_sz: 32768
vfs.zfs.sync_pass_rewrite: 2
vfs.zfs.sync_pass_dont_compress: 5
vfs.zfs.sync_pass_deferred_free: 2
vfs.zfs.zio.dva_throttle_enabled: 1
vfs.zfs.zio.exclude_metadata: 0
vfs.zfs.zio.use_uma: 1
vfs.zfs.zil_slog_limit: 786432
vfs.zfs.cache_flush_disable: 0
vfs.zfs.zil_replay_disable: 0
vfs.zfs.min_auto_ashift: 12
vfs.zfs.max_auto_ashift: 13
vfs.zfs.vdev.trim_max_pending: 10000
vfs.zfs.vdev.bio_delete_disable: 0
vfs.zfs.vdev.bio_flush_disable: 0
vfs.zfs.vdev.queue_depth_pct: 1000
vfs.zfs.vdev.write_gap_limit: 4096
vfs.zfs.vdev.read_gap_limit: 32768
vfs.zfs.vdev.aggregation_limit: 131072
vfs.zfs.vdev.trim_max_active: 64
vfs.zfs.vdev.trim_min_active: 1
vfs.zfs.vdev.scrub_max_active: 2
vfs.zfs.vdev.scrub_min_active: 1
vfs.zfs.vdev.async_write_max_active: 10
vfs.zfs.vdev.async_write_min_active: 1
vfs.zfs.vdev.async_read_max_active: 3
vfs.zfs.vdev.async_read_min_active: 1
vfs.zfs.vdev.sync_write_max_active: 10
vfs.zfs.vdev.sync_write_min_active: 10
vfs.zfs.vdev.sync_read_max_active: 10
vfs.zfs.vdev.sync_read_min_active: 10
vfs.zfs.vdev.max_active: 1000
vfs.zfs.vdev.async_write_active_max_dirty_percent: 60
vfs.zfs.vdev.async_write_active_min_dirty_percent: 30
vfs.zfs.vdev.mirror.non_rotating_seek_inc: 1
vfs.zfs.vdev.mirror.non_rotating_inc: 0
vfs.zfs.vdev.mirror.rotating_seek_offset: 1048576
vfs.zfs.vdev.mirror.rotating_seek_inc: 5
vfs.zfs.vdev.mirror.rotating_inc: 0
vfs.zfs.vdev.trim_on_init: 1
vfs.zfs.vdev.cache.bshift: 16
vfs.zfs.vdev.cache.size: 0
vfs.zfs.vdev.cache.max: 16384
vfs.zfs.vdev.metaslabs_per_vdev: 200
vfs.zfs.txg.timeout: 5
vfs.zfs.space_map_blksz: 4096
vfs.zfs.spa_min_slop: 134217728
vfs.zfs.spa_slop_shift: 5
vfs.zfs.spa_asize_inflation: 24
vfs.zfs.deadman_enabled: 1
vfs.zfs.deadman_checktime_ms: 5000
vfs.zfs.deadman_synctime_ms: 1000000
vfs.zfs.debug_flags: 0
vfs.zfs.debugflags: 0
vfs.zfs.recover: 0
vfs.zfs.spa_load_verify_data: 1
vfs.zfs.spa_load_verify_metadata: 1
vfs.zfs.spa_load_verify_maxinflight: 10000
vfs.zfs.ccw_retry_interval: 300
vfs.zfs.check_hostid: 1
vfs.zfs.mg_fragmentation_threshold: 85
vfs.zfs.mg_noalloc_threshold: 0
vfs.zfs.condense_pct: 200
vfs.zfs.metaslab.bias_enabled: 1
vfs.zfs.metaslab.lba_weighting_enabled: 1
vfs.zfs.metaslab.fragmentation_factor_enabled: 1
vfs.zfs.metaslab.preload_enabled: 1
vfs.zfs.metaslab.preload_limit: 3
vfs.zfs.metaslab.unload_delay: 8
vfs.zfs.metaslab.load_pct: 50
vfs.zfs.metaslab.min_alloc_size: 33554432
vfs.zfs.metaslab.df_free_pct: 4
vfs.zfs.metaslab.df_alloc_threshold: 131072
vfs.zfs.metaslab.debug_unload: 0
vfs.zfs.metaslab.debug_load: 0
vfs.zfs.metaslab.fragmentation_threshold: 70
vfs.zfs.metaslab.gang_bang: 16777217
vfs.zfs.free_bpobj_enabled: 1
vfs.zfs.free_max_blocks: 18446744073709551615
vfs.zfs.no_scrub_prefetch: 0
vfs.zfs.no_scrub_io: 0
vfs.zfs.resilver_min_time_ms: 3000
vfs.zfs.free_min_time_ms: 1000
vfs.zfs.scan_min_time_ms: 1000
vfs.zfs.scan_idle: 50
vfs.zfs.scrub_delay: 4
vfs.zfs.resilver_delay: 2
vfs.zfs.top_maxinflight: 32
vfs.zfs.zfetch.array_rd_sz: 1048576
vfs.zfs.zfetch.max_idistance: 67108864
vfs.zfs.zfetch.max_distance: 8388608
vfs.zfs.zfetch.min_sec_reap: 2
vfs.zfs.zfetch.max_streams: 8
vfs.zfs.prefetch_disable: 0
vfs.zfs.delay_scale: 500000
vfs.zfs.delay_min_dirty_percent: 60
vfs.zfs.dirty_data_sync: 67108864
vfs.zfs.dirty_data_max_percent: 10
vfs.zfs.dirty_data_max_max: 4294967296
vfs.zfs.dirty_data_max: 853010841
vfs.zfs.max_recordsize: 1048576
vfs.zfs.send_holes_without_birth_time: 1
vfs.zfs.mdcomp_disable: 0
vfs.zfs.nopwrite_enabled: 1
vfs.zfs.dedup.prefetch: 1
vfs.zfs.l2c_only_size: 0
vfs.zfs.mfu_ghost_data_esize: 393216
vfs.zfs.mfu_ghost_metadata_esize: 224032256
vfs.zfs.mfu_ghost_size: 224425472
vfs.zfs.mfu_data_esize: 0
vfs.zfs.mfu_metadata_esize: 2973696
vfs.zfs.mfu_size: 8696320
vfs.zfs.mru_ghost_data_esize: 261177344
vfs.zfs.mru_ghost_metadata_esize: 0
vfs.zfs.mru_ghost_size: 261177344
vfs.zfs.mru_data_esize: 6841171968
vfs.zfs.mru_metadata_esize: 9506816
vfs.zfs.mru_size: 6954673152
vfs.zfs.anon_data_esize: 0
vfs.zfs.anon_metadata_esize: 0
vfs.zfs.anon_size: 22016
vfs.zfs.l2arc_norw: 1
vfs.zfs.l2arc_feed_again: 1
vfs.zfs.l2arc_noprefetch: 1
vfs.zfs.l2arc_feed_min_ms: 200
vfs.zfs.l2arc_feed_secs: 1
vfs.zfs.l2arc_headroom: 2
vfs.zfs.l2arc_write_boost: 8388608
vfs.zfs.l2arc_write_max: 8388608
vfs.zfs.arc_meta_limit: 1803964416
vfs.zfs.arc_free_target: 14080
vfs.zfs.compressed_arc_enabled: 1
vfs.zfs.arc_shrink_shift: 7
vfs.zfs.arc_average_blocksize: 8192
vfs.zfs.arc_min: 901982208
vfs.zfs.arc_max: 7215857664

> Finally when its performing well can you repeat the gstat -pd
>

dT: 1.001s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
    3   3887   3887 426514    0.7      0      0    0.0      0      0    0.0   90.7| ada0
    3   3987   3987 434702    0.7      0      0    0.0      0      0    0.0   92.0| ada1

dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s   kBps   ms/d   %busy Name
    3   3958   3958 433563    0.7      0      0    0.0      0      0    0.0   91.6| ada0
    3   3989   3989 438417    0.7      0      0    0.0      0      0    0.0   93.0| ada1

test at f111beta2:~ # dd if=/testdb/test of=/dev/null bs=1m
16000+0 records in
16000+0 records out
16777216000 bytes transferred in 19.385855 secs (865435959 bytes/sec)

Note that this is with vfs.zfs.arc_min and vfs.zfs.arc_max unconstrained.  In previous testing on FreeBSD 10.3 Stable, I had tried a setting of "4000M" for vfs.zfs.arc_min and vfs.zfs.arc_max and had still experienced the performance degradation though that was using the GEOM ELI layering so this latest testing using ashift instead on FreeBSD 11.1 Beta 2 may exhibit a different behavior.

--
Aaron
This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. PLEASE NOTE that all incoming e-mails sent to Weatherford e-mail accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional e-mails (spam). This process could result in deletion of a legitimate e-mail before it is read by its intended recipient at our organization. Moreover, based on the scanning results, the full text of e-mails and attachments may be made available to Weatherford security and other personnel for review and appropriate action. If you have any concerns about this process, please contact us at dataprivacy at weatherford.com.