zpool - low speed write
Alex V. Petrov
alexvpetrov at gmail.com
Thu Aug 5 10:49:26 UTC 2010
>
> Can you please remove use of the zpool entirely (e.g. zpool destroy
> tank) and do a write test to each disk itself? E.g.:
>
> dd if=/dev/zero of=/dev/ad8 bs=64k count=1000000
> dd if=/dev/zero of=/dev/ad10 bs=64k count=1000000
> dd if=/dev/zero of=/dev/ad12 bs=64k count=1000000
I don't have free space for move my data from zpool (1,91T)
> I don't recommend using large block sizes (e.g. bs=1M, bs=3M).
dd if=/dev/zero of=/tank/test.zero bs=64k count=10000
10000+0 records in
10000+0 records out
655360000 bytes transferred in 50.294832 secs (13030365 bytes/sec)
> If all of the above dds show good/decent throughput, then there's
> something strange going on with ZFS. If this is the case, I would
> recommend filing a PR and posting to freebsd-fs about the problem,
> pointing folks to this thread.
>
> If all of the dds show bad throughput, then could you please do the
> following:
>
> - Provide vmstat -i output
vmstat -i output
interrupt total rate
irq1: atkbd0 2368 0
irq6: fdc0 17 0
irq16: vgapci0 ath+ 1728264 100
irq18: uhci2 ehci0* 2183829 127
irq19: uhci4+ 427434 24
irq21: uhci1 42295 2
irq23: uhci3 ehci1 18154 1
cpu0: timer 34317326 1997
irq256: hdac0 1561005 90
irq257: re0 2458465 143
cpu1: timer 34316042 1997
cpu3: timer 34316081 1997
cpu2: timer 34316130 1997
Total 145687410 8482
> - Install ports/sysutils/smartmontools and run smartctl -a /dev/ad8,
> smartctl -a /dev/ad10, and smartctl -a /dev/ad12
In the first message I wrote that smatmontools installed
smartd daily output:
Checking health of /dev/ada2: OK
Checking health of /dev/ada3: OK
Checking health of /dev/ada4: OK
In the logs there are no any error messages that the controller and drives
smartctl -a /dev/ad8
smartctl 5.39.1 2010-01-28 r3054 [FreeBSD 8.1-STABLE amd64] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green family
Device Model: WDC WD10EADS-00M2B0
Serial Number: WD-WCAV51709425
Firmware Version: 01.00A01
User Capacity: 1 000 204 886 016 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Aug 5 18:42:22 2010 KRAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 25) The self-test routine was aborted by
the host.
Total time to complete Offline
data collection: (20400) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off
support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 235) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always
- 0
3 Spin_Up_Time 0x0027 122 109 021 Pre-fail Always
- 6875
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always
- 72
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always
- 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always
- 0
9 Power_On_Hours 0x0032 093 093 000 Old_age Always
- 5561
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 41
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always
- 35
193 Load_Cycle_Count 0x0032 184 184 000 Old_age Always
- 49237
194 Temperature_Celsius 0x0022 110 104 000 Old_age Always
- 37
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always
- 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline
- 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours)
LBA_of_first_error
# 1 Extended offline Aborted by host 90% 5414 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
**********
smartctl -a /dev/ad10
smartctl 5.39.1 2010-01-28 r3054 [FreeBSD 8.1-STABLE amd64] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green family
Device Model: WDC WD10EADS-00L5B1
Serial Number: WD-WCAU4D726772
Firmware Version: 01.01A01
User Capacity: 1 000 204 886 016 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Aug 5 18:43:16 2010 KRAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting
command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine
completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (24600) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off
support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 255) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always
- 0
3 Spin_Up_Time 0x0027 165 161 021 Pre-fail Always
- 6741
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always
- 24
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always
- 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always
- 0
9 Power_On_Hours 0x0032 098 098 000 Old_age Always
- 2103
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 18
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always
- 6
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always
- 24
194 Temperature_Celsius 0x0022 115 108 000 Old_age Always
- 35
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always
- 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline
- 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
**********
smartctl -a /dev/ad12
smartctl 5.39.1 2010-01-28 r3054 [FreeBSD 8.1-STABLE amd64] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Green family
Device Model: WDC WD10EADS-00M2B0
Serial Number: WD-WMAV50095864
Firmware Version: 01.00A01
User Capacity: 1 000 204 886 016 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Thu Aug 5 18:45:25 2010 KRAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x84) Offline data collection activity
was suspended by an interrupting
command from host.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine
completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (20760) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off
support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 239) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x303f) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED
WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always
- 0
3 Spin_Up_Time 0x0027 120 108 021 Pre-fail Always
- 6958
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always
- 117
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always
- 0
7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always
- 0
9 Power_On_Hours 0x0032 088 088 000 Old_age Always
- 9125
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always
- 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always
- 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always
- 63
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always
- 9
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always
- 91
194 Temperature_Celsius 0x0022 111 102 000 Old_age Always
- 36
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always
- 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always
- 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline
- 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always
- 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline
- 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
> If only one of the dds shows bad throughput, then please:
>
> - Install ports/sysutils/smartmontools and run smartctl -a /dev/XXX,
> where XXX is the disk which has bad throughput
> - Try making a ZFS pool with all 3 disks, but then do "zpool offline
> tank XXX" and then re-attempt the following dd:
> dd if=/dev/zero of=/tank/test.zero bs=64k count=1000000
> And see what throughput looks like.
>
> Thanks.
-----
Alex V. Petrov
More information about the freebsd-stable
mailing list