kern/147890: [ufs] [regression] ufs-related lock problem in RELENG_8 (18.04.2010 -> 20.04.2010)

[Kostik Belousov]

On Thu, Jun 17, 2010 at 10:28:41AM +0300, Kostik Belousov wrote:
> On Wed, Jun 16, 2010 at 10:20:03PM +0000, Dmitry Pryanishnikov wrote:
> > The following reply was made to PR kern/147890; it has been noted by GNATS.
> > 
> > From: Dmitry Pryanishnikov <lynx.ripe at gmail.com>
> > To: Bruce Cran <bruce at cran.org.uk>
> > Cc: bug-followup at freebsd.org
> > Subject: Re: kern/147890: [ufs] [regression] ufs-related lock problem in 
> > 	RELENG_8 (18.04.2010 -&gt; 20.04.2010)
> > Date: Thu, 17 Jun 2010 01:12:17 +0300
> > 
> >  Hello!
> >  
> >  2010/6/16 Bruce Cran <bruce at cran.org.uk>:
> >  > Could you run the commands shown in
> >  > http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug-deadlocks.html
> >  > to get more information to help debug the problem please?
> >  
> >    Done - I've rebuilt the kernel from the sources as of 20.04.2010
> >  with the missing debugging options, repeated the problem, turned
> I suspect you do not. In particular, DEBUG_VFS_LOCKS, DEBUG_LOCKS
> and WITNESS seems to be missed.
> 
> >  ddb(4) capture buffer on, issued recommended ddb commands, then called
> >  'doadump'; then rebooted and extracted capture buffer contents from
> >  the crashdump using ddb(8).
> Due to missed options, we cannot trace the lock chains and see why the
> issue happens.
> 
> >  
> >    BTW, is there a way to get this information _without repeating the
> >  problem_ - using only a crashdump from it's previous occurrence? All
> >  data required for analysis should already be there. I suspect not
> >  every deadlock is easy repeatable (like this one)...
> It is, but you have to understand how locks work, and what you are
> looking for.
> 
> >  
> >  db> ps
> >  
> >    pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
> >   2143  1846  2143     0  S+      ufs      0xcd466dc8 sync
> >   2131     1  1911  1000  S       ufs      0xcd466dc8 initial thread
> ...
> >     18     0     0     0  SL      ufs      0xcd466dc8 [syncer]
> ....
> >  Tracing command syncer pid 18 tid 100067 td 0xc7383000
> >  sched_switch(c7383000,0,104,191,e660f504,...) at 0xc05292c0 = sched_switch+0x330
> >  mi_switch(104,0,c0745097,1eb,50,...) at 0xc0510680 = mi_switch+0x200
> >  sleepq_switch(c7383000,0,c0745097,260,50,...) at 0xc054028f =
> >  sleepq_switch+0x15f
> >  sleepq_wait(cd466dc8,50,c07393ba,4,0,...) at 0xc0540f63 = sleepq_wait+0x63
> >  __lockmgr_args(cd466dc8,80100,cd466e30,0,0,...) at 0xc04f087f =
> >  __lockmgr_args+0xb1f
> >  ffs_lock(e732cb34,c054944b,c0763314,80100,cd466d70,...) at 0xc067e9f1
> >  = ffs_lock+0xa1
> >  VOP_LOCK1_APV(c07a47c0,e732cb34,c73830a4,c07b7a20,cd466d70,...) at
> >  0xc070c945 = VOP_LOCK1_APV+0xb5
> >  _vn_lock(cd466d70,80100,c074e6ce,82b,4,...) at 0xc05a2fd8 = _vn_lock+0x78
> >  vget(cd466d70,80100,c7383000,39a,c763b870,...) at 0xc05969db = vget+0xbb
> >  qsync(c763b870,0,c0761f41,552,4,...) at 0xc0687db7 = qsync+0x197
> >  ffs_sync(c763b870,3,c074e6ce,d6b,c763b870,...) at 0xc067a139 = ffs_sync+0x349
> >  sync_fsync(e732cc7c,c0772c36,c76b8778,e732cc7c,c76b86b8,...) at
> >  0xc059812f = sync_fsync+0x18f
> >  VOP_FSYNC_APV(c0799bc0,e732cc7c,c074e6ce,6a5,c7383000,...) at
> >  0xc070a545 = VOP_FSYNC_APV+0xc5
> >  sync_vnode(c093b618,c093b604,3e8,6d4,4e20,...) at 0xc059840b = sync_vnode+0x16b
> >  sched_sync(0,e732cd38,c073bc0b,343,c759f7f8,...) at 0xc0598753 =
> >  sched_sync+0x273
> >  fork_exit(c05984e0,0,e732cd38) at 0xc04dddf8 = fork_exit+0xb8
> >  fork_trampoline() at 0xc06d6b80 = fork_trampoline+0x8
> >  --- trap 0, eip = 0, esp = 0xe732cd70, ebp = 0 ---
> Do you have custoom kernel build, in particular, are quotas compiled in ?
> What are the drives ?
> 
> Having to pull the generic information about the problematic system
> does not make debugging faster.

What _could_ happen in your case is the relock in ufs_accessx for QUOTA.
ufs_delete_denied() calls VOP_ACCESSX for both parent directory
and target vnode while parent directory is locked. If source directory
is not exclusively locked, LOR can happen, since ufs_accessx has to
unlock and lock directory vnode. I am not sure how many similar cases
of VOP_ACCESS[X] is places in the problematic locations.

Try the (untested) patch below.

diff --git a/sys/ufs/ufs/ufs_lookup.c b/sys/ufs/ufs/ufs_lookup.c
index 0030c52..c779491 100644
--- a/sys/ufs/ufs/ufs_lookup.c
+++ b/sys/ufs/ufs/ufs_lookup.c
@@ -77,6 +77,32 @@ SYSCTL_INT(_debug, OID_AUTO, dircheck, CTLFLAG_RW, &dirchk, 0, "");
 /* true if old FS format...*/
 #define OFSFMT(vp)	((vp)->v_mount->mnt_maxsymlinklen <= 0)
 
+#ifdef QUOTA
+static int
+ufs_lookup_upgrade_lock(struct vnode *vp)
+{
+	int error;
+
+	ASSERT_VOP_LOCKED(vp, __FUNCTION__);
+	if (VOP_ISLOCKED(vp) == LK_EXCLUSIVE)
+		return (0);
+
+	error = 0;
+
+	/*
+	 * Upgrade vnode lock, since getinoquota()
+	 * requires exclusive lock to modify inode.
+	 */
+	vhold(vp);
+	vn_lock(vp, LK_UPGRADE | LK_RETRY);
+	VI_LOCK(vp);
+	if (vp->v_iflag & VI_DOOMED)
+		error = ENOENT;
+	vdropl(vp);
+	return (error);
+}
+#endif
+
 static int
 ufs_delete_denied(struct vnode *vdp, struct vnode *tdp, struct ucred *cred,
     struct thread *td)
@@ -232,6 +258,13 @@ ufs_lookup_ino(struct vnode *vdp, struct vnode **vpp, struct componentname *cnp,
 	vnode_create_vobject(vdp, DIP(dp, i_size), cnp->cn_thread);
 
 	bmask = VFSTOUFS(vdp->v_mount)->um_mountp->mnt_stat.f_iosize - 1;
+#ifdef QUOTA
+	if ((nameiop == DELETE || nameiop == RENAME) && (flags & ISLASTCN)) {
+		error = ufs_lookup_upgrade_lock(vdp);
+		if (error != 0)
+			return (error);
+	}
+#endif
 
 restart:
 	bp = NULL;
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20100617/27626188/attachment.pgp