Adaptec 3210S, 4.9-STABLE, corruption when disk fails

[Uwe Doering]

Don Bowman wrote:
> I have a machine running:
> 
> $ uname -a
> FreeBSD machine.phaedrus.sandvine.com 4.9-STABLE FreeBSD 4.9-STABLE #0:
> Fri Mar 19 10:39:07 EST 2004
> user at machine.phaedrus.sandvine.com:/usr/src/sys/compile/LABDB  i386
> 
> It has an adaptec 3210S raid controller running a single raid-5, and
> runs postgresql 7.4.6 as its primary application.
> 
> 3 times now I have had a drive fail, and have had corrupted files in the
> postgresql cluster @ the same time.
> 
> The time is too closely correlated to be a coincidence. It passes fsck @
> the time that I got to it a couple of hours later, and the filesystem
> seems to be ok (with a failed drive, the raid in 'degrade' mode).
> 
> It appears that the drive failure and the postgresql failure occur @
> exactly the same time (monitoring with nagios, within 1hr accuracy). It
> would appear that for some file(s) bad data was returned.
> 
> Does anyone have any suggestions?

In my experience, in a situation like this RAID controllers can block 
the system for up to a couple of minutes, trying to revive a failed disk 
drive by sending it bus reset commands and the like, until they 
eventually give up and drop into degraded mode.  With sufficiently 
patient applications this is no problem, but if a program runs into 
internal timeouts during this period of time bad things can happen.

My point is that while the disk controller may trigger the problem the 
instance that actually corrupts the database might be PostgreSQL itself. 
  Of course, I'm aware that it's going to be quite hard to tell for sure 
who the culprit is.

    Uwe
-- 
Uwe Doering         |  EscapeBox - Managed On-Demand UNIX Servers
gemini at geminix.org  |  http://www.escapebox.net