panic/lock on 9.3-RELEASE with nullfs/nfs/zfs combination

[Harald Schmalzbauer]

 Bezüglich Konstantin Belousov's Nachricht vom 24.07.2014 18:59
(localtime):
> …
>> panic: LK_RETRY set with incompatible flags (0x200400) or an error
>> occured (11)
>> cpuid = 3
>> KDB: stack backtrace:
>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame
>> 0xffffff82e54bcc70
>> kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e54bcd30
>> panic() at panic+0x1cd/frame 0xffffff82e54bce30
>> _vn_lock() at _vn_lock+0x67/frame 0xffffff82e54bce90
>> zfs_lookup() at zfs_lookup+0x420/frame 0xffffff82e54bcf20
>> zfs_freebsd_lookup() at zfs_freebsd_lookup+0xa6/frame 0xffffff82e54bd070
>> VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xd8/frame 0xffffff82e54bd0a0
>> vfs_cache_lookup() at vfs_cache_lookup+0xff/frame 0xffffff82e54bd110
>> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xd8/frame 0xffffff82e54bd140
>> null_lookup() at null_lookup+0x92/frame 0xffffff82e54bd1c0
>> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xd8/frame 0xffffff82e54bd1f0
>> lookup() at lookup+0x389/frame 0xffffff82e54bd290
>> namei() at namei+0x3df/frame 0xffffff82e54bd340
>> vn_open_cred() at vn_open_cred+0x1e2/frame 0xffffff82e54bd4b0
>> vop_stdvptocnp() at vop_stdvptocnp+0x1af/frame 0xffffff82e54bd7e0
>> null_vptocnp() at null_vptocnp+0xf5/frame 0xffffff82e54bd850
>> VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0xdb/frame 0xffffff82e54bd880
>> vn_vptocnp_locked() at vn_vptocnp_locked+0x15b/frame 0xffffff82e54bd910
>> vn_fullpath1() at vn_fullpath1+0x100/frame 0xffffff82e54bd970
>> kern___getcwd() at kern___getcwd+0xd4/frame 0xffffff82e54bd9d0
>> amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e54bdaf0
>> Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e54bdaf0
>> --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x8011a191c, rsp =
>> 0x7fffffffe658, rbp = 0x801873400 ---
>> KDB: enter: panic
>> [ thread pid 1905 tid 100856 ]
>> Stopped at kdb_enter+0x3b: movq $0,0x642172(%rip)
>>
>> Like mentioned, this panic happens only if a nfs(v4) client visits fs15
>> (the exported and nullfs_mounted fs) and I try to rw-open any file on
>> the nullfs afterwards!!!
>>
>> How can I provide useful info with KDB? I don't have a dumpdev available
>> in that machine???
>> http://www.es.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html
>> seems not applicaple, no /var/crash/?*???
>>
> The lockmgr flags are LK_SHARE | LK_RETRY, and error 11 == EDEADLK
> indicates that the lock is already taken by the curthread in the
> exclusive mode. I am interested in what line of code did the locking.
>
> Add ddb, INVARIANTS, WITNESS and DEBUG_VFS_LOCKS options to the kernel
> config, reproduce the issue and, after the panic occured and you
> get at the ddb prompt, issue command 'show alllocks'.
>
> Also, do 'show mount', after which do 'show mount <addr>', where <addr>
> is the address of your nullfs mount point, printed by 'show mount'.
>
> I need all console output starting from the panic message.

Maybe it's of any use for you:
(kgdb) backtrace
#0 doadump (textdump=1) at
/usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/kern_shutdown.c:271
#1 0xffffffff804aa198 in kern_reboot (howto=260) at
/usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/kern_shutdown.c:454
#2 0xffffffff804a9bd5 in panic (fmt=0x104 <Address 0x104 out of bounds>)
at /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/kern_shutdown.c:642
#3 0xffffffff8055dcd7 in _vn_lock (vp=0xfffffe002dc119d8, flags=2098176,
file=<value optimized out>, line=<value optimized out>)
at /usr/local/share/deploy-tools/RELENG_9_3/src/sys/kern/vfs_vnops.c:1402
#4 0xffffffff80fb8c90 in u8_decomp_b4_tbl () from /boot/kernel/zfs.ko
#5 0xfffffe0007ac0b00 in ?? ()
#6 0x0000000081051480 in ?? ()
#7 0xffffff82e5660598 in ?? ()
#8 0xffffff82e5660180 in ?? ()
#9 0x0000000081050c88 in ?? ()
#10 0xfffffe002d37c0e8 in ?? ()
#11 0xffffffff80a8f940 in sdt_vfs_vop_vop_unlock_return ()
#12 0xfffffe002dc119d8 in ?? ()
#13 0x0000000900000000 in ?? ()
#14 0x000000002dcaa0c0 in ?? ()
#15 0x000001e8e565ff20 in ?? ()
#16 0xffffff82e565ff40 in ?? ()
#17 0xffffff82e5660598 in ?? ()
#18 0xffffff82e56600b0 in ?? ()
#19 0xffffff82e5660180 in ?? ()
#20 0xffffff82e56600b0 in ?? ()
#21 0xffffff82e5660070 in ?? ()
#22 0xffffffff80fb8e36 in u8_decomp_b4_tbl () from /boot/kernel/zfs.ko
#23 0xfffffe00ba802920 in ?? ()
#24 0xfffffe0000000000 in ?? ()
#25 0x0000000000002e2e in ?? ()
#26 0x0000000000000000 in ?? ()
(kgdb) up 10
#10 0xfffffe002d37c0e8 in ?? ()
(kgdb) list
1402 KASSERT((flags & LK_RETRY) == 0 || error == 0,
1403 ("LK_RETRY set with incompatible flags (0x%x) or an error occured
(%d)",
1404 flags, error));
1405 /*
1406 * Callers specify LK_RETRY if they wish to get dead vnodes.
1407 * If RETRY is not set, we return ENOENT instead.
1408 */
1409 if (error == 0 && vp->v_iflag & VI_DOOMED &&
1410 (flags & LK_RETRY) == 0) {
1411 VOP_UNLOCK(vp, 0);

Just tellme if you need anything else, hav vmcore in kgdb available!

Thanks,

-Harry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 196 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140724/5cdb8dfb/attachment.sig>