File system corruption with ATA RAID-1 on 6-STABLE
O. Hartmann
ohartman at mail.uni-mainz.de
Fri Dec 30 02:18:33 PST 2005
Seán C. Farley schrieb:
> On Wed, 21 Dec 2005, Brian Fundakowski Feldman wrote:
>
>> On Sat, Nov 19, 2005 at 12:06:00PM -0600, Seán C. Farley wrote:
>>
>>> I installed 6.0 on a system with two SATA drives in RAID-1. 5.4 is
>>> installed on a standard IDE channel. All appeared to work well until
>>> sometime after updating the system to 6-STABLE, installing the Nvidia
>>> driver and copying a lot of data (mainly ogg files) from my old hard
>>> drive onto it. I started seeing a multitude of READ_DMA (or maybe
>>> WRITE_DMA) errors in /var/log/messages. The system was beginning to
>>> slow down a lot, so I rebooted. Since then I have been unable to
>>> keep it up without a panic after an fsck in single-user mode.
>>>
>>> I realize this does not tell much, but just when ld.so.hints is being
>>> removed due to being corrupt the follow panic appears: Panic:
>>> ffs_valloc: dup alloc The panic may just be from a system that is too
>>> corrupted. I am amazed it still boots.
>>
>>
>> That's how it appears, yeah.
>>
>> What does the drive health look like? Seems most like a chipset bug
>> or nvidia driver bug, but could be things are just coinciding with
>> drive or drive controller failure.
>
>
> I have since reinstalled and have been unable to reproduce the problem.
> 6-STABLE as of November 28 works without any obvious problems. I have
> also used the Nvidia driver v8174 and v8178.
>
> I do see this in messages (only one):
> Dec 16 02:31:46 thor kernel: ad6: TIMEOUT - WRITE_DMA retrying (1 retry
> left) LBA=195511730
>
> I am also attaching the information about my SATA drives (from
> smartctl). The first drive (ad4) has three errors logged.
>
> Seán
>
>
> ------------------------------------------------------------------------
>
> smartctl version 5.33 [i386-portbld-freebsd6.0] Copyright (C) 2002-4 Bruce Allen
> Home page is http://smartmontools.sourceforge.net/
>
> === START OF INFORMATION SECTION ===
> Device Model: ST3160827AS
> Serial Number: 4MT0L7EY
> Firmware Version: 3.42
> User Capacity: 160,041,885,696 bytes
> Device is: Not in smartctl database [for details use: -P showall]
> ATA Version is: 6
> ATA Standard is: ATA/ATAPI-6 T13 1410D revision 2
> Local Time is: Thu Dec 29 15:08:26 2005 CST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> General SMART Values:
> Offline data collection status: (0x82) Offline data collection activity
> was completed without error.
> Auto Offline Data Collection: Enabled.
> Self-test execution status: ( 0) The previous self-test routine completed
> without error or no self-test has ever
> been run.
> Total time to complete Offline
> data collection: ( 430) seconds.
> Offline data collection
> capabilities: (0x5b) SMART execute Offline immediate.
> Auto Offline data collection on/off support.
> Suspend Offline collection upon new
> command.
> Offline surface scan supported.
> Self-test supported.
> No Conveyance Self-test supported.
> Selective Self-test supported.
> SMART capabilities: (0x0003) Saves SMART data before entering
> power-saving mode.
> Supports SMART auto save timer.
> Error logging capability: (0x01) Error logging supported.
> No General Purpose Logging support.
> Short self-test routine
> recommended polling time: ( 1) minutes.
> Extended self-test routine
> recommended polling time: ( 94) minutes.
>
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x000f 057 046 006 Pre-fail Always - 66158687
> 3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0
> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 3
> 5 Reallocated_Sector_Ct 0x0033 099 099 036 Pre-fail Always - 46
> 7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 210711557
> 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3673
> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 3
> 194 Temperature_Celsius 0x0022 030 040 000 Old_age Always - 30 (Lifetime Min/Max 0/21)
> 195 Hardware_ECC_Recovered 0x001a 057 046 000 Old_age Always - 66158687
> 197 Current_Pending_Sector 0x0012 100 099 000 Old_age Always - 0
> 198 Offline_Uncorrectable 0x0010 100 099 000 Old_age Offline - 0
> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
> 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
> 202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0
>
> SMART Error Log Version: 1
> ATA Error Count: 3
> CR = Command Register [HEX]
> FR = Features Register [HEX]
> SC = Sector Count Register [HEX]
> SN = Sector Number Register [HEX]
> CL = Cylinder Low Register [HEX]
> CH = Cylinder High Register [HEX]
> DH = Device/Head Register [HEX]
> DC = Device Command Register [HEX]
> ER = Error register [HEX]
> ST = Status register [HEX]
> Powered_Up_Time is measured from power on, and printed as
> DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
> SS=sec, and sss=millisec. It "wraps" after 49.710 days.
>
> Error 3 occurred at disk power-on lifetime: 2904 hours (121 days + 0 hours)
> When the command that caused the error occurred, the device was active or idle.
>
> After command completion occurred, registers were:
> ER ST SC SN CL CH DH
> -- -- -- -- -- -- --
> 40 51 00 1f b1 67 e8 Error: UNC at LBA = 0x0867b11f = 141013279
>
> Commands leading to the command that caused the error were:
> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
> -- -- -- -- -- -- -- -- ---------------- --------------------
> c8 00 20 1f b1 67 e8 00 22:51:33.459 READ DMA
> ca 00 08 eb 07 c1 e4 00 22:51:33.459 WRITE DMA
> c6 00 10 00 00 00 e0 00 22:51:33.459 SET MULTIPLE MODE
> ef 02 00 00 00 00 e0 00 22:51:33.459 SET FEATURES [Enable write cache]
> ef aa 00 00 00 00 e0 00 22:51:33.459 SET FEATURES [Enable read look-ahead]
>
> Error 2 occurred at disk power-on lifetime: 2904 hours (121 days + 0 hours)
> When the command that caused the error occurred, the device was active or idle.
>
> After command completion occurred, registers were:
> ER ST SC SN CL CH DH
> -- -- -- -- -- -- --
> 40 51 00 1f b1 67 e8 Error: UNC at LBA = 0x0867b11f = 141013279
>
> Commands leading to the command that caused the error were:
> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
> -- -- -- -- -- -- -- -- ---------------- --------------------
> c8 00 20 1f b1 67 e8 00 22:51:33.459 READ DMA
> ca 00 04 5b 04 c1 e4 00 22:51:33.459 WRITE DMA
> c6 00 10 00 00 00 e0 00 22:51:33.459 SET MULTIPLE MODE
> ef 02 00 00 00 00 e0 00 22:51:33.459 SET FEATURES [Enable write cache]
> ef aa 00 00 00 00 e0 00 22:51:33.459 SET FEATURES [Enable read look-ahead]
>
> Error 1 occurred at disk power-on lifetime: 2904 hours (121 days + 0 hours)
> When the command that caused the error occurred, the device was active or idle.
>
> After command completion occurred, registers were:
> ER ST SC SN CL CH DH
> -- -- -- -- -- -- --
> 40 51 00 1f b1 67 e8 Error: UNC at LBA = 0x0867b11f = 141013279
>
> Commands leading to the command that caused the error were:
> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
> -- -- -- -- -- -- -- -- ---------------- --------------------
> c8 00 20 1f b1 67 e8 00 22:51:28.346 READ DMA
> ca 00 0c df 07 c1 e4 00 22:51:28.345 WRITE DMA
> ca 00 14 5f b9 76 e3 00 22:51:28.342 WRITE DMA
> ca 00 20 df 9c 75 e3 00 22:51:28.334 WRITE DMA
> ca 00 20 1f 32 ba ec 00 22:51:28.333 WRITE DMA
>
> SMART Self-test log structure revision number 1
> Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
> # 1 Short offline Completed without error 00% 3672 -
>
> SMART Selective self-test log data structure revision number 1
> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> 1 0 0 Not_testing
> 2 0 0 Not_testing
> 3 0 0 Not_testing
> 4 0 0 Not_testing
> 5 0 0 Not_testing
> Selective self-test flags (0x0):
> After scanning selected spans, do NOT read-scan remainder of disk.
> If Selective self-test is pending on power-up, resume after 0 minute delay.
>
>
>
> ------------------------------------------------------------------------
>
> smartctl version 5.33 [i386-portbld-freebsd6.0] Copyright (C) 2002-4 Bruce Allen
> Home page is http://smartmontools.sourceforge.net/
>
> === START OF INFORMATION SECTION ===
> Device Model: ST3160827AS
> Serial Number: 4MT0LNFT
> Firmware Version: 3.42
> User Capacity: 160,041,885,696 bytes
> Device is: Not in smartctl database [for details use: -P showall]
> ATA Version is: 6
> ATA Standard is: ATA/ATAPI-6 T13 1410D revision 2
> Local Time is: Thu Dec 29 15:08:30 2005 CST
> SMART support is: Available - device has SMART capability.
> SMART support is: Enabled
>
> === START OF READ SMART DATA SECTION ===
> SMART overall-health self-assessment test result: PASSED
>
> General SMART Values:
> Offline data collection status: (0x82) Offline data collection activity
> was completed without error.
> Auto Offline Data Collection: Enabled.
> Self-test execution status: ( 0) The previous self-test routine completed
> without error or no self-test has ever
> been run.
> Total time to complete Offline
> data collection: ( 430) seconds.
> Offline data collection
> capabilities: (0x5b) SMART execute Offline immediate.
> Auto Offline data collection on/off support.
> Suspend Offline collection upon new
> command.
> Offline surface scan supported.
> Self-test supported.
> No Conveyance Self-test supported.
> Selective Self-test supported.
> SMART capabilities: (0x0003) Saves SMART data before entering
> power-saving mode.
> Supports SMART auto save timer.
> Error logging capability: (0x01) Error logging supported.
> No General Purpose Logging support.
> Short self-test routine
> recommended polling time: ( 1) minutes.
> Extended self-test routine
> recommended polling time: ( 94) minutes.
>
> SMART Attributes Data Structure revision number: 10
> Vendor Specific SMART Attributes with Thresholds:
> ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
> 1 Raw_Read_Error_Rate 0x000f 063 052 006 Pre-fail Always - 89874069
> 3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0
> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 3
> 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
> 7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 215371066
> 9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3673
> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 3
> 194 Temperature_Celsius 0x0022 030 040 000 Old_age Always - 30 (Lifetime Min/Max 0/20)
> 195 Hardware_ECC_Recovered 0x001a 063 052 000 Old_age Always - 89874069
> 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
> 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
> 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
> 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
> 202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0
>
> SMART Error Log Version: 1
> No Errors Logged
>
> SMART Self-test log structure revision number 1
> Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
> # 1 Short offline Completed without error 00% 3672 -
>
> SMART Selective self-test log data structure revision number 1
> SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
> 1 0 0 Not_testing
> 2 0 0 Not_testing
> 3 0 0 Not_testing
> 4 0 0 Not_testing
> 5 0 0 Not_testing
> Selective self-test flags (0x0):
> After scanning selected spans, do NOT read-scan remainder of disk.
> If Selective self-test is pending on power-up, resume after 0 minute delay.
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> freebsd-stable at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
I had a similar problem with a FreeBSD 5.4. Suddenly, one of two drive,
a Samsung SpinPoint SP2004C began to produce wild errors, then, after a
while, everything seems to be clear. I also did a SMART self test,
without revealing any obscure matter.
At the end, suspecting everything except the cabling, I changed the
cabling and that's it! I do not know why and how, the cable has no
obvious damage (but maybe broken copper wires). It's a simple hint, but
test it, it is maybe worth testing it.
SATA cabling is really a worse cheap type of cabling. The idea behind is
good, but the 'how they made is reality' is a horror!
O.
More information about the freebsd-stable
mailing list