UFS2 corruption (RELENG_7, amd64)

[Chris Dillon]

Quoting Oliver Fromme <olli at lurza.secnetix.de>:

...snip...

> # fsck -y /dev/da45
> ** /dev/da45
> ** Last Mounted on [...]
> ** Phase 1 - Check Blocks and Sizes
> ** Phase 2 - Check Pathnames
> UNALLOCATED  I=2107428  OWNER=2283830233 MODE=0
> SIZE=0 MTIME=Feb  5 08:43 2008
> NAME=/D.0131ae2e/B.0786
>
> UNEXPECTED SOFT UPDATE INCONSISTENCY
>
> REMOVE? yes

I've been seeing "UNEXPECTED SOFT UPDATE INCONSISTENCY" fsck errors  
show up in some of my filesytem snapshots on a RELENG_6 AMD64 box for  
years (actually since RELENG_5), which will eventually lead to "panic:  
snapblkfree: inconsistent block type" if those snapshots are mounted  
and used.  There are no SCSI or "bad block" errors or any error of any  
other kind that shows up on my system before the problems occur.  I  
think I used default newfs settings when creating the (comparatively  
small) 300GB filesystem.  The problem happens rarely and on a  
production system so I've never been able to pin it down to anything  
in particular.

I've found that others have seen the same panic, likely from the same cause:

http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/115645 (see first  
reply in Audit-Trail)

http://lists.freebsd.org/pipermail/freebsd-stable/2006-August/027916.html
http://lists.freebsd.org/pipermail/freebsd-stable/2006-August/027917.html

I did very recently determine that the "snapblkfree: inconsistent  
block type" panic I see is due directly to the "soft update  
inconsistency" in the snapshots, so there is some filesystem  
corruption happening somewhere which leads to the panic later.  I have  
been regularly fscking the snapshots and deleting any of them that  
have errors so they won't panic the system later.  It has kept the  
system panic-free for quite some time now.

A commonality  you will notice amongst all our systems is that they  
are AMD64.  I've been unable to find anyone having this problem on i386.

-- 

Chris Dillon - NetEng/SysAdm
Reeds Spring R-IV School District
Technology Department
175 Elementary Rd.
Reeds Spring, MO  65737
Voice: 417-272-8266   Fax: 417-272-0015