Strange UFS write problem & SU+J "unexpected inconsistences" on 9.1-STABLE r253105 after it on OTHER filesystems.

[Lev Serebryakov]

Hello, freebsd-fs.

  My server paniced tonight with UFS problem:

ufs/root[WRITE(offset=385499136, length=16384)]error = 22
g_vfs_done():ufs/root[WRITE(offset=385564672, length=16384)]error = 22
g_vfs_done():ufs/root[WRITE(offset=385712128, length=16384)]error = 22
g_vfs_done():ufs/root[WRITE(offset=385826816, length=16384)]error = 22
g_vfs_done():ufs/root[WRITE(offset=770703360, length=16384)]error = 22
g_vfs_done():ufs/root[WRITE(offset=770719744, length=16384)]error = 22
g_vfs_done():ufs/var[WRITE(offset=268539904, length=2048)]error = 22
/var: got error 22 while accessing filesystem
panic: softdep_deallocate_dependencies: unrecovered I/O error
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff8047a836 at kdb_backtrace+0x66
#1 0xffffffff8044382e at panic+0x1ce
#2 0xffffffff8059c040 at clear_remove+0
#3 0xffffffff804bf835 at brelse+0x75
#4 0xffffffff804c2258 at bufdone+0x68
#5 0xffffffff804bcb0e at biodone+0xae
#6 0xffffffff803e289c at g_io_schedule_up+0xac
#7 0xffffffff803e2ffc at g_up_procbody+0x5c
#8 0xffffffff804144ef at fork_exit+0x11f
#9 0xffffffff805f53de at fork_trampoline+0xe

  and "fsck_ffs" refused to fix two OTHER (/usr and /tmp) SU+J-enabled FFSes with same messages:

Journal file sequence mismatch XXX != YYY
UNEXPECTED SU+J INCONSISTENCY
INTERNAL ERROR: GOT TO reply()
UNIXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY

and exited with signal 11.

 So, here are two questions:

(1) What does "error 22" mean? Disk doesn't show ANY errors in S.M.A.R.T.
(and all internal tests are Ok). Also, here are NO ANY driver (AHCI) errors
in post-mortem dump. It doesn't look like hardware problem.

(2) How to avoid fsck refuses in such situations? Why OTHER (not ones with
write errors) FSes get errors? It looks like one another problem with SU+J.

 Please note, these FSes reside directly on SATA drive, without any software or
hardware RAIDs.

 I have dumped both FSes with "dumpfs" and "dumpfs -f" before manual check
and have block-dumped /tmp (as it is small enough).

 You could find them at

 http://lev.serebryakov.spb.ru/FreeBSD/suj-crash/



-- 
// Black Lion AKA Lev Serebryakov <lev at FreeBSD.org>