panic: lockmgr on FreeBSD 7.0-RELEASE-p4 amd64

[John Baldwin]

On Thursday 25 September 2008 01:34:06 am Jeff Wheelhouse wrote:
> 
> On Sep 24, 2008, at 12:34 PM, John Baldwin wrote:
> 
> > On Wednesday 24 September 2008 12:17:56 pm Jeff Wheelhouse wrote:
> >> panic: lockmgr: thread 0xffffff0050858350, not exclusive lock holder
> >> 0xffffff00074959f0 unlocking
> >> cpuid = 0
> >> KDB: stack backtrace:
> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> >> panic() at panic+0x17a
> >> _lockmgr() at _lockmgr+0x872
> >> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
> >> null_unlock() at null_unlock+0xff
> >> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
> >> nullfs_mount() at nullfs_mount+0x244
> >> vfs_donmount() at vfs_donmount+0xe4d
> >> nmount() at nmount+0xa5
> >> syscall() at syscall+0x254
> >> Xfast_syscall() at Xfast_syscall+0xab
> >> --- syscall (378, FreeBSD ELF64, nmount), rip = 0x206845ac, rsp =
> >> 0x7fffffffdfc8, rbp = 0x7fffffffdfd0 ---
> >
> > Can you use gdb or the like to get the souce file/line for the
> > nullfs_mount+0x244 frame?
> 
> Got it again, this time with the full debug kernel, and I'm getting  
> the same weird results from gdb, so I'll go ahead and post it:
> 
> panic: lockmgr: thread 0xffffff0003e499f0, not exclusive lock holder  
> 0xffffff000a5e16a0 unlocking
> cpuid = 0
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> panic() at panic+0x17a
> _lockmgr() at _lockmgr+0x872
> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
> null_unlock() at null_unlock+0xff
> VOP_UNLOCK_APV() at VOP_UNLOCK_APV+0x46
> nullfs_mount() at nullfs_mount+0x244
> vfs_donmount() at vfs_donmount+0xe4d
> nmount() at nmount+0xa5
> syscall() at syscall+0x254
> Xfast_syscall() at Xfast_syscall+0xab
> --- syscall (378, FreeBSD ELF64, nmount), rip = 0x206845ac, rsp =  
> 0x7fffffffe1c8, rbp = 0x7fffffffe1d0 ---
> 
> $ gdb /boot/kernel/nullfs.ko
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and  
> you are
> welcome to change it and/or distribute copies of it under certain  
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for  
> details.
> This GDB was configured as "amd64-marcel-freebsd"...
> (gdb) l *nullfs_mount+0x244
> 0x9c4 is in nullfs_mount (namei.h:163).
> 158		struct thread *td)
> 159	{
> 160		ndp->ni_cnd.cn_nameiop = op;
> 161		ndp->ni_cnd.cn_flags = flags;
> 162		ndp->ni_segflg = segflg;
> 163		ndp->ni_dirp = namep;
> 164		ndp->ni_cnd.cn_thread = td;
> 165	}
> 166	
> 167	#define NDF_NO_DVP_RELE		0x00000001
> (gdb)
> 
> (That's NDINIT(), but line 163 doesn't look like it belongs in the  
> middle of a call stack.  There's a VOP_UNLOCK a few lines above  
> NDINIT() in mount_nullfs(), and another one some ways farther on in  
> the function.)

It's probably the one just before the NDINIT (note that the return address in 
the call stack is pointing to the next instruction to be executed after the 
call to VOP_UNLOCK(), so sometimes it can end up referring to the next line 
in the source code from the actual function call):

        if ((mp->mnt_vnodecovered->v_op == &null_vnodeops) &&
                VOP_ISLOCKED(mp->mnt_vnodecovered)) {
                VOP_UNLOCK(mp->mnt_vnodecovered, 0);
                isvnunlocked = 1;
        }
        /*
         * Find lower node
         */
        NDINIT(ndp, LOOKUP, FOLLOW|LOCKLEAF,
                UIO_SYSSPACE, target, td);
        error = namei(ndp);

Can you 'p *mp'?  I'm curious if mp->mnt_vnodecovered is NULL (in which case, 
why didn't the two tests in the if() fail?)

> The good news is we took this particular machine out of production and  
> came up with a synthetic test based on our in-house code that can  
> probably reliably reproduce this within a few minutes.  As you might  
> expect, the test involves hammering the same nullfs mount point with  
> mounts and umounts from multiple processes without any external  
> synchronization.

Ok.  Reproducibility is good. :)

-- 
John Baldwin