Strange UFS write problem & SU+J "unexpected inconsistences" on 9.1-STABLE r253105 after it on OTHER filesystems.

[Edward Tomasz Napierała]

Wiadomość napisana przez Lev Serebryakov <lev at FreeBSD.org> w dniu 21 wrz 2013, o godz. 12:48:
> Hello, freebsd-fs.
> 
>  My server paniced tonight with UFS problem:
> 
> ufs/root[WRITE(offset=385499136, length=16384)]error = 22
> g_vfs_done():ufs/root[WRITE(offset=385564672, length=16384)]error = 22
> g_vfs_done():ufs/root[WRITE(offset=385712128, length=16384)]error = 22
> g_vfs_done():ufs/root[WRITE(offset=385826816, length=16384)]error = 22
> g_vfs_done():ufs/root[WRITE(offset=770703360, length=16384)]error = 22
> g_vfs_done():ufs/root[WRITE(offset=770719744, length=16384)]error = 22
> g_vfs_done():ufs/var[WRITE(offset=268539904, length=2048)]error = 22
> /var: got error 22 while accessing filesystem
> panic: softdep_deallocate_dependencies: unrecovered I/O error
> cpuid = 0
> KDB: stack backtrace:
> #0 0xffffffff8047a836 at kdb_backtrace+0x66
> #1 0xffffffff8044382e at panic+0x1ce
> #2 0xffffffff8059c040 at clear_remove+0
> #3 0xffffffff804bf835 at brelse+0x75
> #4 0xffffffff804c2258 at bufdone+0x68
> #5 0xffffffff804bcb0e at biodone+0xae
> #6 0xffffffff803e289c at g_io_schedule_up+0xac
> #7 0xffffffff803e2ffc at g_up_procbody+0x5c
> #8 0xffffffff804144ef at fork_exit+0x11f
> #9 0xffffffff805f53de at fork_trampoline+0xe
> 
>  and "fsck_ffs" refused to fix two OTHER (/usr and /tmp) SU+J-enabled FFSes with same messages:
> 
> Journal file sequence mismatch XXX != YYY
> UNEXPECTED SU+J INCONSISTENCY
> INTERNAL ERROR: GOT TO reply()
> UNIXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY
> 
> and exited with signal 11.
> 
> So, here are two questions:
> 
> (1) What does "error 22" mean? Disk doesn't show ANY errors in S.M.A.R.T.
> (and all internal tests are Ok). Also, here are NO ANY driver (AHCI) errors
> in post-mortem dump. It doesn't look like hardware problem.

According to "man errno", 22 is EINVAL.  Which is... weird, to say the least.
What's below the filesystem?  I mean, what's the GEOM tree?

> (2) How to avoid fsck refuses in such situations? Why OTHER (not ones with
> write errors) FSes get errors? It looks like one another problem with SU+J.

Try to make fsck _not_ use journal.  Simply respond "n" when it asks if you want
it to use journal.

> Please note, these FSes reside directly on SATA drive, without any software or
> hardware RAIDs.

No labels, for example?

[..]