7.2 filesystem corruption

[Charles Sprickman]

Hello all,

Not sure where to go with this post, I've tried -fs and -scsi previously 
in trying to track down some panics in the softdep stuff.  Perhaps the 
more general audience here can shove me in the right direction.

I have a box (Dell PE 2970) running FreeBSD 7.2/amd-64.  6 GB of ECC RAM, 
and a Dell-branded LSI RAID controller (mpt driver).  It's a mail server 
with the active mail server running in a jail and a test version of same 
running in another jail (qmail/vpopmail/courier on old, 
postfix/pfadmin/dovecot on new).

It passed a few weeks of heavy stress testing where I was putting much 
more load on it using an imap/pop/smtp test suite before going into 
production with only one panic (which happened during a fairly intense 
mstone run) - I figured I was somewhat on the bleeding edge with 7.x 
64-bit at that time, so I was not overly concerned since I've run into 
softdep panics before.  Since then however, there have been a few panics 
in "ufsdirhash_lookup".

When this happens, the box reboots, does a background fsck and does not 
complain about anything.  I decided background fsck was probably not a 
good idea, so I disabled it and manually fsck'd on all subsequent panics. 
The pattern is similar to this example:

** /dev/mfid0s1g
** Last Mounted on /spool
** Phase 1 - Check Blocks and Sizes
UNKNOWN FILE TYPE I=147718184
UNEXPECTED SOFT UPDATE INCONSISTENCY
CLEAR? yes

PARTIALLY ALLOCATED INODE I=147718185
UNEXPECTED SOFT UPDATE INCONSISTENCY

And in phase 2, lots of this:

UNALLOCATED I=152688468 OWNER=root MODE=0 SIZE=0 MTIME=Dec 31 19:00 1969 
NAME=/jails/mailbak.blah.net/home/vpopmail/domains/blah.net/A/spec/Maildir/new/1233549930.73014.blah.bway.net

UNEXPECTED SOFT UPDATE INCONSISTENCY
REMOVE? yes

And in Phase 4, lots of this:

** Phase 4 - Check Reference Counts
UNREF FILE I=147623979  OWNER=88 MODE=100600
SIZE=0 MTIME=Feb  7 00:19 2010
CLEAR? yes

In the manual runs, I tend to run through about 3 or 4 times, since even 
though the filesystem gets marked "clean", another run finds more errors. 
Once I get two clean runs in a row, I let the box boot.

Regardless of how "clean" the fs is, I have consistently seen messages 
like this in my serial console log:

g_vfs_done():mfid0s1g[READ(offset=2456998070156636160, length=16384)]error 
= 5
g_vfs_done():mfid0s1g[READ(offset=2456998070156636160, length=16384)]error 
= 5

On the last run, I also turned off soft updates for good measure.

Now I occasinally get these errors:

g_vfs_done():mfid0s1g[READ(offset=5335388948596480000, length=16384)]error 
= 5
bad block 838May 18 00:29:14 8bigmail kernel: 3pid 24481 (rm), 0uid 0 
inumber 1571657736 on /spoo6l: bad block
76548920427, ino 151657736

In addition, there are some files that now have bizarre flags set, such as 
"schg", "sappnd", "opaque", etc.  Some can be changed, others give a "bad 
file descriptor" error.

I fear the fs is getting more scrambled.

I started to think that I'm probably dealing with two things - some bug in 
64-bit UFS2, plus a perpetually dirty filesystem that causes the box to 
panic, which causes more corruption, and so on.

I do have the option of trying to schedule a huge maintenance window and 
dumping the fs, newfs'ing it, and then restoring it, but it's a tough sell 
and for various reasons I can't put a ton of time into this (anyone that 
knows me, hit me up offlist for a fun story).  I'm also quite concerned 
that fsck is finding and fixing things, but the fs is still obviously not 
quite "right".  In short, how can I ensure this won't happen a week after 
a dump/restore?

So that's the story, here's my questions:

-Is there any interest in tracking down what the nature of the initial 
panic/corruption is?  I know I'm a release behind, but digging through the 
PR database, nothing stuck out as far as softdep, mpt, or dirhash bugs 
that looked similar to what I'm seeing that got fixed in 7.3.

-Where is the most likely place to look for a problem here?  The mpt 
driver?  The megacli utility and the bios utility both claim the array is 
in great shape.  The only fs that ever shows the errors with "g_vfs_done" 
and the nonsensical offsets is the partition where the jails reside.  Or 
is it ufsdirhash thing?  I saw some interesting bug reports, but nothing 
that quite matched.  UFS2/SU itself?

-If I do dump/restore (or pull from backups), should I stick to 7.2 or go 
to 7.3 while I'm working on the box?  Or gamble on 8.0 (where I've oddly 
enough seen much fewer odd thigns of late)?

For reference, here's a few other queries regarding this issue:

http://marc.info/?l=freebsd-stable&m=125901173424554&w=2
http://old.nabble.com/7.2-p4:-panic:-ufsdirhash_lookup:-bad-offset-in-hash-array-td27715632.html

I still have some core dumps sitting here as well.

Any input would be appreciated - I do have more info available, but this 
message is already about twice as long as I'd like it to be.  Hit me up 
with any questions.

Thanks,

Charles