File system corruption with ATA RAID-1 on 6-STABLE
Seán C. Farley
sean-freebsd at farley.org
Thu Dec 29 13:15:23 PST 2005
On Wed, 21 Dec 2005, Brian Fundakowski Feldman wrote:
> On Sat, Nov 19, 2005 at 12:06:00PM -0600, Seán C. Farley wrote:
>> I installed 6.0 on a system with two SATA drives in RAID-1. 5.4 is
>> installed on a standard IDE channel. All appeared to work well until
>> sometime after updating the system to 6-STABLE, installing the Nvidia
>> driver and copying a lot of data (mainly ogg files) from my old hard
>> drive onto it. I started seeing a multitude of READ_DMA (or maybe
>> WRITE_DMA) errors in /var/log/messages. The system was beginning to
>> slow down a lot, so I rebooted. Since then I have been unable to
>> keep it up without a panic after an fsck in single-user mode.
>>
>> I realize this does not tell much, but just when ld.so.hints is being
>> removed due to being corrupt the follow panic appears: Panic:
>> ffs_valloc: dup alloc The panic may just be from a system that is too
>> corrupted. I am amazed it still boots.
>
> That's how it appears, yeah.
>
> What does the drive health look like? Seems most like a chipset bug
> or nvidia driver bug, but could be things are just coinciding with
> drive or drive controller failure.
I have since reinstalled and have been unable to reproduce the problem.
6-STABLE as of November 28 works without any obvious problems. I have
also used the Nvidia driver v8174 and v8178.
I do see this in messages (only one):
Dec 16 02:31:46 thor kernel: ad6: TIMEOUT - WRITE_DMA retrying (1 retry
left) LBA=195511730
I am also attaching the information about my SATA drives (from
smartctl). The first drive (ad4) has three errors logged.
Seán
--
sean-freebsd at farley.org
-------------- next part --------------
smartctl version 5.33 [i386-portbld-freebsd6.0] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Device Model: ST3160827AS
Serial Number: 4MT0L7EY
Firmware Version: 3.42
User Capacity: 160,041,885,696 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 6
ATA Standard is: ATA/ATAPI-6 T13 1410D revision 2
Local Time is: Thu Dec 29 15:08:26 2005 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 430) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 94) minutes.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 057 046 006 Pre-fail Always - 66158687
3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 3
5 Reallocated_Sector_Ct 0x0033 099 099 036 Pre-fail Always - 46
7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 210711557
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3673
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 3
194 Temperature_Celsius 0x0022 030 040 000 Old_age Always - 30 (Lifetime Min/Max 0/21)
195 Hardware_ECC_Recovered 0x001a 057 046 000 Old_age Always - 66158687
197 Current_Pending_Sector 0x0012 100 099 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 099 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0
SMART Error Log Version: 1
ATA Error Count: 3
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 3 occurred at disk power-on lifetime: 2904 hours (121 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 1f b1 67 e8 Error: UNC at LBA = 0x0867b11f = 141013279
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 20 1f b1 67 e8 00 22:51:33.459 READ DMA
ca 00 08 eb 07 c1 e4 00 22:51:33.459 WRITE DMA
c6 00 10 00 00 00 e0 00 22:51:33.459 SET MULTIPLE MODE
ef 02 00 00 00 00 e0 00 22:51:33.459 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 e0 00 22:51:33.459 SET FEATURES [Enable read look-ahead]
Error 2 occurred at disk power-on lifetime: 2904 hours (121 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 1f b1 67 e8 Error: UNC at LBA = 0x0867b11f = 141013279
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 20 1f b1 67 e8 00 22:51:33.459 READ DMA
ca 00 04 5b 04 c1 e4 00 22:51:33.459 WRITE DMA
c6 00 10 00 00 00 e0 00 22:51:33.459 SET MULTIPLE MODE
ef 02 00 00 00 00 e0 00 22:51:33.459 SET FEATURES [Enable write cache]
ef aa 00 00 00 00 e0 00 22:51:33.459 SET FEATURES [Enable read look-ahead]
Error 1 occurred at disk power-on lifetime: 2904 hours (121 days + 0 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 1f b1 67 e8 Error: UNC at LBA = 0x0867b11f = 141013279
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
c8 00 20 1f b1 67 e8 00 22:51:28.346 READ DMA
ca 00 0c df 07 c1 e4 00 22:51:28.345 WRITE DMA
ca 00 14 5f b9 76 e3 00 22:51:28.342 WRITE DMA
ca 00 20 df 9c 75 e3 00 22:51:28.334 WRITE DMA
ca 00 20 1f 32 ba ec 00 22:51:28.333 WRITE DMA
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 3672 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
-------------- next part --------------
smartctl version 5.33 [i386-portbld-freebsd6.0] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
=== START OF INFORMATION SECTION ===
Device Model: ST3160827AS
Serial Number: 4MT0LNFT
Firmware Version: 3.42
User Capacity: 160,041,885,696 bytes
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: 6
ATA Standard is: ATA/ATAPI-6 T13 1410D revision 2
Local Time is: Thu Dec 29 15:08:30 2005 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 430) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 94) minutes.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 063 052 006 Pre-fail Always - 89874069
3 Spin_Up_Time 0x0003 099 099 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 3
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 083 060 030 Pre-fail Always - 215371066
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 3673
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 3
194 Temperature_Celsius 0x0022 030 040 000 Old_age Always - 30 (Lifetime Min/Max 0/20)
195 Hardware_ECC_Recovered 0x001a 063 052 000 Old_age Always - 89874069
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 3672 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
More information about the freebsd-stable
mailing list