How to speed up slow zpool scrub?

[Miroslav Lachman]

Jeremy Faulkner wrote on 04/26/2016 17:59:

> Check the drive health with smartctl (part of sysutils/smartmontools).
> Are the drives desktop drives or nas drives? In gstat is one drive
> busier than the rest? If so, replace that drive.

# iostat -x -w 15 ada0 ada1 ada2 ada3
..
..
                         extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0     215.3  15.7  5045.6   492.5    2   8.1  64
ada1     220.1  15.6  5063.2   492.0    2   8.0  65
ada2     221.7  15.1  4917.9   479.2    1   8.1  66
ada3     224.1  15.2  4910.4   482.9    0   8.1  67
                         extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0     220.8  13.5  4497.7   624.8    0   6.7  61
ada1     224.7  13.1  4503.6   625.3    0   6.8  61
ada2     206.5  12.3  4287.3   599.7    0   6.2  60
ada3     208.3  12.8  4289.2   619.2    0   6.4  60
                         extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0     236.5   8.1  5219.4   392.2    1   6.4  61
ada1     231.5   8.1  5199.4   390.6    1   6.4  61
ada2     242.5   7.5  5075.7   379.7    2   5.4  58
ada3     240.3   7.7  5048.2   383.7    2   5.9  58

Drives are 60% busy when scrubbing and deleting old files are both 
running so I think there is still something to tune.

Miroslav Lachman


# ~/bin/arc_summary.sh
System Memory:
          Physical RAM:  4962 MB
          Free Memory :  108 MB

ARC Size:
          Current Size:             3373 MB (arcsize)
          Target Size (Adaptive):   3372 MB (c)
          Min Size (Hard Limit):    512 MB (zfs_arc_min)
          Max Size (Hard Limit):    3584 MB (zfs_arc_max)

ARC Size Breakdown:
          Most Recently Used Cache Size:          93%    3161 MB (p)
          Most Frequently Used Cache Size:         6%    210 MB (c-p)

ARC Efficency:
          Cache Access Total:             2605059457
          Cache Hit Ratio:      79%       2072273907     [Defined State 
for buffer]
          Cache Miss Ratio:     20%       532785550      [Undefined 
State for Buffer]
          REAL Hit Ratio:       72%       1898031795     [MRU/MFU Hits Only]

          Data Demand   Efficiency:    91%
          Data Prefetch Efficiency:     1%

         CACHE HITS BY CACHE LIST:
           Anon:                        7%        158713873 
  [ New Customer, First Cache Hit ]
           Most Recently Used:         55%        1154441329 (mru) 
  [ Return Customer ]
           Most Frequently Used:       35%        743590466 (mfu) 
  [ Frequent Customer ]
           Most Recently Used Ghost:    0%        11032454 (mru_ghost) 
  [ Return Customer Evicted, Now Back ]
           Most Frequently Used Ghost:  0%        4495785 (mfu_ghost) 
  [ Frequent Customer Evicted, Now Back ]
         CACHE HITS BY DATA TYPE:
           Demand Data:                10%        226614173
           Prefetch Data:               0%        519707
           Demand Metadata:            80%        1669964501
           Prefetch Metadata:           8%        175175526
         CACHE MISSES BY DATA TYPE:
           Demand Data:                 3%        20563634
           Prefetch Data:               9%        50227920
           Demand Metadata:            79%        422689982
           Prefetch Metadata:           7%        39304014
---------------------------------------------