ZFS (or something) is absurdly slow

Eitan Adler lists at eitanadler.com
Sun Nov 24 16:29:37 UTC 2013


On Sun, Nov 24, 2013 at 3:03 AM, Thomas Steen Rasmussen
<thomas at gibfest.dk> wrote:
> On 24-11-2013 04:15, Eitan Adler wrote:
>>
>>
>>> vfsstat.d https://forums.freebsd.org/showpost.php?p=182070&postcount=6
>>
>> I can run this script, what output should I be looking for?
>
>
> Check the sample output on the page: It shows two lists, "Number
> of operations" and "Bytes read or write". The lists are ordered
> with the busiest at the bottom and seperated by filesystem
> location. While things are slow, try running it to see if some
> location on the filesystem is being hammered.....

I will do so.

> Another thing you should probably do is run a SMART check on the
> disk to see if something is wrong with it.

See the complete output below.  The only thing which stands out to me is:
200 Multi_Zone_Error_Rate   0x002a   100   100   000    Old_age
Always       -       19275

>  I had another case with
> a zfs mirror that performed appalingly, turned out it was because
> one of the disks was dodgy, not in a way that made zfs show
> checksum errors, but enough to make it really really slow. Since a
> ZFS vdev only performs as good as the slowest disk in a vdev,
> which in turn will slow the whole pool down, replacing the disk
> made everything much better.

===============

smartctl 6.2 2013-07-26 r3841 [FreeBSD 11.0-CURRENT amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Momentus SpinPoint M8 (AF)
Device Model:     ST1000LM024 HN-M101MBB
Serial Number:    S2U5J9FCB79134
LU WWN Device Id: 5 0004cf 20904e7cf
Firmware Version: 2AR10001
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 3.0, 3.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Nov 24 11:23:21 2013 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (12780) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   2) minutes.
Extended self-test routine
recommended polling time: ( 213) minutes.
SCT capabilities:       (0x003f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   051    Pre-fail
Always       -       25
  2 Throughput_Performance  0x0026   252   252   000    Old_age
Always       -       0
  3 Spin_Up_Time            0x0023   089   089   025    Pre-fail
Always       -       3453
  4 Start_Stop_Count        0x0032   100   100   000    Old_age
Always       -       158
  5 Reallocated_Sector_Ct   0x0033   252   252   010    Pre-fail
Always       -       0
  7 Seek_Error_Rate         0x002e   252   252   051    Old_age
Always       -       0
  8 Seek_Time_Performance   0x0024   252   252   015    Old_age
Offline      -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age
Always       -       7285
 10 Spin_Retry_Count        0x0032   252   252   051    Old_age
Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age
Always       -       280
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age
Always       -       180
191 G-Sense_Error_Rate      0x0022   100   100   000    Old_age
Always       -       155
192 Power-Off_Retract_Count 0x0022   252   252   000    Old_age
Always       -       0
194 Temperature_Celsius     0x0002   055   047   000    Old_age
Always       -       45 (Min/Max 18/63)
195 Hardware_ECC_Recovered  0x003a   100   100   000    Old_age
Always       -       0
196 Reallocated_Event_Count 0x0032   252   252   000    Old_age
Always       -       0
197 Current_Pending_Sector  0x0032   252   252   000    Old_age
Always       -       0
198 Offline_Uncorrectable   0x0030   252   252   000    Old_age
Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age
Always       -       0
200 Multi_Zone_Error_Rate   0x002a   100   100   000    Old_age
Always       -       19275
223 Load_Retry_Count        0x0032   100   100   000    Old_age
Always       -       280
225 Load_Cycle_Count        0x0032   091   091   000    Old_age
Always       -       95664

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has
ever been run
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Completed [00% left] (0-65535)
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

=====================

> What does diskinfo -ct /dev/whatever say about the seek times on the
> bad disk ?

[10005 root at gravity (100%) /home/eitan !2!]#diskinfo -ct /dev/ada1
/dev/ada1
        512             # sectorsize
        1000204886016   # mediasize in bytes (932G)
        1953525168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        1938021         # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        S2U5J9FCB79134  # Disk ident.

I/O command overhead:
        time to read 10MB block      0.099713 sec       =    0.005 msec/sector
        time to read 20480 sectors   1.447996 sec       =    0.071 msec/sector
        calculated command overhead                     =    0.066 msec/sector

Seek times:
        Full stroke:      250 iter in   8.036950 sec =   32.148 msec
        Half stroke:      250 iter in   5.463750 sec =   21.855 msec
        Quarter stroke:   500 iter in  10.542506 sec =   21.085 msec
        Short forward:    400 iter in   5.707363 sec =   14.268 msec
        Short backward:   400 iter in   4.645333 sec =   11.613 msec
        Seq outer:       2048 iter in   0.096977 sec =    0.047 msec
        Seq inner:       2048 iter in   1.853596 sec =    0.905 msec
Transfer rates:
        outside:       102400 kbytes in   0.949048 sec =   107898 kbytes/sec
        middle:        102400 kbytes in   1.659245 sec =    61715 kbytes/sec
        inside:        102400 kbytes in   2.020322 sec =    50685 kbytes/sec


> Are the results the same if you boot off of an usb stick and
> test the disk when it is completely idle and independent of the running OS ?

Good question.  I can not check this at the moment.



-- 
Eitan Adler


More information about the freebsd-fs mailing list