vfs deadlock during panic?

[Jeremy Chadwick]

On Thu, Feb 25, 2010 at 12:14:04AM -0800, Jake Holland wrote:
> > All the backtraces indicate UFS snapshots are being used, as in 
> > you're doing dump -L or mksnap_ffs.  Is that the case?
> 
> Not to my knowledge, but I'm not sure of everything that's happening
> during startup.  The 3 I listed last time appear in the serial console,
> usually within 10s of the login prompt's first appearance after reboot.
> If I do more file operations, I typically see more such messages, but
> not every time.
> 
> For example, I tried running umount on my /var/crash (mounted on
> /dev/mfid0s3d) and it gave me this:
> 
> lock order reversal:
>  1st 0xffffff00104337e8 ufs (ufs) @ kern/vfs_mount.c:1204
>  2nd 0xffffff001045aa58 devfs (devfs) @ ufs/ffs/ffs_vfsops.c:1194
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x81e
> __lockmgr_args() at __lockmgr_args+0xd31
> vop_stdlock() at vop_stdlock+0x39
> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b
> _vn_lock() at _vn_lock+0x57
> ffs_flushfiles() at ffs_flushfiles+0xc5
> softdep_flushfiles() at softdep_flushfiles+0x63
> ffs_unmount() at ffs_unmount+0x2e1
> dounmount() at dounmount+0x2e6
> unmount() at unmount+0x27e
> syscall() at syscall+0x102
> Xfast_syscall() at Xfast_syscall+0xe1
> --- syscall (22, FreeBSD ELF64, unmount), rip = 0x80069f4bc, rsp =
> 0x7fffffffe2f
> 8, rbp = 0 ---
> 
> But I got nothing when I ran mount -a afterwards.
> 
> 
> Then I ran vi on /etc/ports.conf and saw this:
> 
> lock order reversal:
>  1st 0xffffff81983bd5b8 bufwait (bufwait) @ kern/vfs_bio.c:2559
>  2nd 0xffffff000bd2ac00 dirhash (dirhash) @ ufs/ufs/ufs_dirhash.c:285
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> _witness_debugger() at _witness_debugger+0x2e
> witness_checkorder() at witness_checkorder+0x81e
> _sx_xlock() at _sx_xlock+0x55
> ufsdirhash_acquire() at ufsdirhash_acquire+0x44
> ufsdirhash_add() at ufsdirhash_add+0x19
> ufs_direnter() at ufs_direnter+0x88b
> ufs_makeinode() at ufs_makeinode+0x2a7
> VOP_CREATE_APV() at VOP_CREATE_APV+0xb3
> vn_open_cred() at vn_open_cred+0x482
> kern_openat() at kern_openat+0x179
> syscall() at syscall+0x102
> Xfast_syscall() at Xfast_syscall+0xe1
> --- syscall (5, FreeBSD ELF64, open), rip = 0x800ce075c, rsp =
> 0x7fffffffe0d8, r
> bp = 0xf ---
> 
> 
> But only the first time.  Subsequently re-opening the same or another
> few arbitrary text files with vi did not produce any repeats.  And on
> some occasions, nothing came up even on the first run of vi, though
> usually the first vi does give a similar stack.

Are you sure one of the filesystems on the disk isn't corrupt?  There's
been reports of this problem in the past, but AFAIR it doesn't manifest
itself in this manner.

Just something to consider: if you have another disk you can put in the
box and install FreeBSD 8.x + upgrade to latest -STABLE on it, and it
doesn't behave this way, then I'd say there's some underlying filesystem
corruption of some kind.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |