panic/lock on 9.3-RELEASE with nullfs/nfs/zfs combination

[Konstantin Belousov]

On Fri, Jul 25, 2014 at 08:59:56AM +0200, Harald Schmalzbauer wrote:
>  Bez?glich Konstantin Belousov's Nachricht vom 24.07.2014 21:30
> (localtime):
> > On Thu, Jul 24, 2014 at 08:28:11PM +0200, Harald Schmalzbauer wrote:
> >>  Bez??glich Konstantin Belousov's Nachricht vom 24.07.2014 18:59
> >> (localtime):
> >>> ???
> >>> The lockmgr flags are LK_SHARE | LK_RETRY, and error 11 == EDEADLK
> >>> indicates that the lock is already taken by the curthread in the
> >>> exclusive mode. I am interested in what line of code did the locking.
> >>>
> >>> Add ddb, INVARIANTS, WITNESS and DEBUG_VFS_LOCKS options to the kernel
> >>> config, reproduce the issue and, after the panic occured and you
> >>> get at the ddb prompt, issue command 'show alllocks'.
> >>>
> >>> Also, do 'show mount', after which do 'show mount <addr>', where <addr>
> >>> is the address of your nullfs mount point, printed by 'show mount'.
> >>>
> >>> I need all console output starting from the panic message.
> >> FreeBSD/amd64 (mira.inop.mo1.omnilan.net) (ttyu0)
> >>
> >> login: panic: LK_RETRY set with incompatible flags (0x200400) or an
> >> error occured (11)
> >> cpuid = 1
> >> KDB: stack backtrace:
> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame
> >> 0xffffff82e565fc70
> >> kdb_backtrace() at kdb_backtrace+0x37/frame 0xffffff82e565fd30
> >> panic() at panic+0x1cd/frame 0xffffff82e565fe30
> >> _vn_lock() at _vn_lock+0x77/frame 0xffffff82e565fe90
> >> zfs_lookup() at zfs_lookup+0x420/frame 0xffffff82e565ff20
> >> zfs_freebsd_lookup() at zfs_freebsd_lookup+0xa6/frame 0xffffff82e5660070
> >> VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x102/frame
> >> 0xffffff82e56600a0
> >> vfs_cache_lookup() at vfs_cache_lookup+0xff/frame 0xffffff82e5660110
> >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x102/frame 0xffffff82e5660140
> >> null_lookup() at null_lookup+0x92/frame 0xffffff82e56601c0
> >> VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x102/frame 0xffffff82e56601f0
> >> lookup() at lookup+0x32f/frame 0xffffff82e5660290
> >> namei() at namei+0x3df/frame 0xffffff82e5660340
> >> vn_open_cred() at vn_open_cred+0x1e2/frame 0xffffff82e56604b0
> >> vop_stdvptocnp() at vop_stdvptocnp+0x1af/frame 0xffffff82e56607e0
> >> null_vptocnp() at null_vptocnp+0xf5/frame 0xffffff82e5660850
> >> VOP_VPTOCNP_APV() at VOP_VPTOCNP_APV+0x105/frame 0xffffff82e5660880
> >> vn_vptocnp_locked() at vn_vptocnp_locked+0x15b/frame 0xffffff82e5660910
> >> vn_fullpath1() at vn_fullpath1+0x100/frame 0xffffff82e5660970
> >> kern___getcwd() at kern___getcwd+0xd4/frame 0xffffff82e56609d0
> >> amd64_syscall() at amd64_syscall+0x318/frame 0xffffff82e5660af0
> >> Xfast_syscall() at Xfast_syscall+0xf7/frame 0xffffff82e5660af0
> >> --- syscall (326, FreeBSD ELF64, sys___getcwd), rip = 0x8011a191c, rsp =
> >> 0x7fffffffe658, rbp = 0x801873400 ---
> >> KDB: enter: panic
> >> [ thread pid 1918 tid 100919 ]
> >> Stopped at kdb_enter+0x3b: movq $0,0x645622(%rip)
> >> db> show alllocks
> >> db> <???nothing
> > You do not have WITNESS in your kernel config, most likely.
> 
> Hmm enabled it is, from the boot msg:
> gcc version 4.2.1 20070831 patched [FreeBSD]
> WARNING: WITNESS option enabled, expect reduced performance.
> CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3292.52-MHz K8-class CPU)
> 
> But there's another WITNESS message:
> WITNESS: unable to allocate a new witness object.
> Ill ask the relatives what they know about this message (aunt bing,
> uncle google aso).

Yes, this is the problem. In sys/kern/subr_witness.c, change the
#define WITNESS_COUNT from 1024 to e.g. 2048.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140725/026049b5/attachment.sig>