Sleeping thread owns a nonsleepable lock panic (& lor)

Kostik Belousov kostikbel at gmail.com
Mon Jul 25 08:59:10 UTC 2011


On Mon, Jul 25, 2011 at 12:21:07PM +0200, Herve Boulouis wrote:
> Hi list,
> 
> We have 2 freebsd 8.2-STABLE (cvsuped june 22) that keeps crashing in a bad way :
> 
> The are doing heavy apache / php4 web serving from a nfs mount and panic at least once a day
> with the following message (no crash dump produced, hand copied from the console) :
> 
> Sleeping on "vmopar" with the following non-sleepable locks held:
> exclusive sleep mutex NFSnode lock (NFSnode lock) r =  0 (0xffffff0201798000) locked @ nfsclient/nfs_subs.c:538
> lock order reversal:
>  1st 0xffffffff018ff6da80 turnstile lock (turnstile lock) @ kern/subr_turnstile.c:190
>  2nd 0xffffffffff80b52b10 scrlock (scrlock) @ dev/syscons.c:2570
> lock order reversal:
>  1st 0xffffffff018ff6da80 turnstile lock (turnstile lock) @ kern/subr_turnstile.c:190
>  2nd 0xffffffffff80b78ef8 sleepq chain (sleepq chain) @ kern/subr_turnstile.c:203
> lock order reversal:
>  1st 0xffffffffff80b78ef8 sleepq chain (sleepq chain) @ kern/subr_turnstile.c:203
>  2nd 0xffffffffff80b52b10 scrlock (scrlock) @ dev/syscons.c:2570
> Sleeping thread (tid 100998, pid 20700) owns a non-sleepable lock
> panic: sleeping thread
> cpuid = 1
> panic: bufwrite: buffer is not busy???
> cpuid = 1
> 
> The 2 servers share the same load and panic consistently. I enabled WITNESS on the 2 in the hope
> it would allow the boxes to auto reboot after panic and get extra debug info. I got debug info
> but the servers still hangs after the double panic :(

Try this. Calling vnode_pager_setsize() while holding a mutex is prohibited.
On the other hand, I remember that my attempt to add a strict assert
that a vnode is exclusively locked in vnode_pager_setsize() had to be
reversed because nfs_loadattrcache() sometimes called without vnode
lock held.

commit 2aa7d15c38b0c01e3f724f04d7ed02ce11c82cc0
Author: Konstantin Belousov <kostikbel at gmail.com>
Date:   Mon Jul 25 11:56:04 2011 +0300

    Postpone the vnode_pager_setsize() call until the nfs node mutex is dropped.

diff --git a/sys/nfsclient/nfs_subs.c b/sys/nfsclient/nfs_subs.c
index 19fde06..351885a 100644
--- a/sys/nfsclient/nfs_subs.c
+++ b/sys/nfsclient/nfs_subs.c
@@ -478,7 +478,9 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp,
 	struct timespec mtime, mtime_save;
 	int v3 = NFS_ISV3(vp);
 	int error = 0;
+	int do_setsize;
 
+	do_setsize = 0;
 	md = *mdp;
 	t1 = (mtod(md, caddr_t) + md->m_len) - *dposp;
 	cp2 = nfsm_disct(mdp, dposp, NFSX_FATTR(v3), t1, M_WAIT);
@@ -606,7 +608,7 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp,
 				np->n_size = vap->va_size;
 				np->n_flag |= NSIZECHANGED;
 			}
-			vnode_pager_setsize(vp, np->n_size);
+			do_setsize = 1;
 		} else {
 			np->n_size = vap->va_size;
 		}
@@ -643,6 +645,8 @@ nfs_loadattrcache(struct vnode **vpp, struct mbuf **mdp, caddr_t *dposp,
 		KDTRACE_NFS_ATTRCACHE_LOAD_DONE(vp, &np->n_vattr, 0);
 #endif
 	mtx_unlock(&np->n_mtx);
+	if (do_setsize)
+		vnode_pager_setsize(vp, np->n_size);
 out:
 #ifdef KDTRACE_HOOKS
 	if (error)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20110725/712cc504/attachment.pgp


More information about the freebsd-stable mailing list