kern/136945: [ufs] [lor] filedesc structure/ufs (poll)
John Baldwin
jhb at freebsd.org
Thu Jul 30 13:52:12 UTC 2009
On Thursday 30 July 2009 5:25:07 am Kostik Belousov wrote:
> On Thu, Jul 30, 2009 at 11:05:32AM +0200, Rene Ladan wrote:
> > 2009/7/29 John Baldwin <jhb at freebsd.org>:
> > > On Wednesday 29 July 2009 11:20:21 am Rene Ladan wrote:
> > >> 2009/7/29 John Baldwin <jhb at freebsd.org>:
> > >> > On Wednesday 29 July 2009 5:52:24 am Rene Ladan wrote:
> > >> >> 2009/7/28 John Baldwin <jhb at freebsd.org>:
> > >> >> > On Tuesday 28 July 2009 10:03:40 am Rene Ladan wrote:
> > >> >> >> 2009/7/28 John Baldwin <jhb at freebsd.org>:
> > >> >> >> > On Monday 27 July 2009 10:00:05 am Rene Ladan wrote:
> > >> >> >> >> The following reply was made to PR kern/136945; it has been
noted
> > > by
> > >> >> > GNATS.
> > >> >> >> >>
> > >> >> >> >> From: Rene Ladan <rene at freebsd.org>
> > >> >> >> >> To: John Baldwin <jhb at freebsd.org>
> > >> >> >> >> Cc: bug-followup at freebsd.org
> > >> >> >> >> Subject: Re: kern/136945: [ufs] [lor] filedesc structure/ufs
(poll)
> > >> >> >> >> Date: Mon, 27 Jul 2009 15:51:15 +0200
> > >> >> >> >>
> > >> >> >> >> 2009/7/27 John Baldwin <jhb at freebsd.org>:
> > >> >> >> >> > I would actually expect this to be the correct order for
these
> > > two
> > >> >> >> > locks.=
> > >> >> >> >> =A0Can
> > >> >> >> >> > you capture the output of the 'debug.witness.fullgraph'
sysctl
> > > to a
> > >> >> > file?
> > >> >> >> >> >
> > >> >> >> >> Yes, see attachment. I'm still running the same 8.0-BETA2.
> > >> >> >> >
> > >> >> >> > Hmm, the attachment was eaten by a grue, can you post the file
> > >> > somewhere?
> > >> >> >> >
> > >> >> >> Yes, see ftp://rene-ladan.nl/pub/freebsd/kern_136945.txt
> > >> >> >
> > >> >> > Ok, it looks like it did encounter a UFS -> filedesc order at some
> > >> > point. Can
> > >> >> > you patch sys/kern/subr_witness.c to add a section to the
order_lists[]
> > >> > array
> > >> >> > after the 'ZFS locking list' and before the spin locks list that
looks
> > >> > like
> > >> >> > this:
> > >> >> >
> > >> >> > { "filedesc structure", &lock_class_sx },
> > >> >> > { "ufs", &lock_class_lockmgr},
> > >> >> > { NULL, NULL },
> > >> >> >
> > >> >> The LOR seems to be gone, previously it showed up only once right
> > >> >> after booting the system.
> > >> >>
> > >> >> But now a new LOR (according to the LOR page) seems pop up:
> > >> >> Trying to mount root from ufs:/dev/ad0s1a
> > >> >> lock order reversal:
> > >> >> 1st 0xffffff0002a4ad80 ufs (ufs)
> > > @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465
> > >> >> 2nd 0xffffff0002b29a48 filedesc structure (filedesc structure) @
> > >> >> /usr/src/sys/kern/kern_descrip.c:2478
> > >> >> KDB: stack backtrace:
> > >> >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> > >> >> _witness_debugger() at _witness_debugger+0x49
> > >> >> witness_checkorder() at witness_checkorder+0x7ea
> > >> >> _sx_xlock() at _sx_xlock+0x44
> > >> >> mountcheckdirs() at mountcheckdirs+0x80
> > >> >> vfs_donmount() at vfs_donmount+0xfbf
> > >> >> kernel_mount() at kernel_mount+0xa1
> > >> >> vfs_mountroot_try() at vfs_mountroot_try+0x177
> > >> >> vfs_mountroot() at vfs_mountroot+0x47d
> > >> >> start_init() at start_init+0x62
> > >> >> fork_exit() at fork_exit+0x12a
> > >> >> fork_trampoline() at fork_trampoline+0xe
> > >> >> --- trap 0, rip = 0, rsp = 0xffffff800001ad30, rbp = 0 ---
> > >> >>
> > >> >> The output of `df' and `mount' looks ok.
> > >> >
> > >> > Yes, this is the "real" LOR as "filedesc" -> "ufs" in the poll() case
> > > should
> > >> > be the normal order. I believe this should fix it. mountcheckdirs()
> > > doesn't
> > >> > need the vnodes locked, it just needs the caller to hold references
on
> > > them
> > >> > so they aren't recycled:
> > >> >
> > >> > --- //depot/projects/smpng/sys/kern/vfs_mount.c#96
> > >> > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c
> > >> > @@ -1069,9 +1069,10 @@
> > >> > vfs_event_signal(NULL, VQ_MOUNT, 0);
> > >> > if (VFS_ROOT(mp, LK_EXCLUSIVE, &newdp))
> > >> > panic("mount: lost mount");
> > >> > + VOP_UNLOCK(newdp, 0);
> > >> > + VOP_UNLOCK(vp, 0);
> > >> > mountcheckdirs(vp, newdp);
> > >> > - vput(newdp);
> > >> > - VOP_UNLOCK(vp, 0);
> > >> > + vrele(newdp);
> > >> > if ((mp->mnt_flag & MNT_RDONLY) == 0)
> > >> > error = vfs_allocate_syncvnode(mp);
> > >> > vfs_unbusy(mp);
> > >> >
> > >> The LOR is still present, but at a different place without the
> > >> mountcheckdirs() call (not on the LOR page either) :
> > >
> > > Ok, try this patch as well:
> > >
> > > --- //depot/projects/smpng/sys/kern/vfs_mount.c#97
> > > +++ /home/jhb/work/p4/smpng/sys/kern/vfs_mount.c
> > > @@ -1481,6 +1481,8 @@
> > > if (VFS_ROOT(TAILQ_FIRST(&mountlist), LK_EXCLUSIVE, &rootvnode))
> > > panic("Cannot find root vnode");
> > >
> > > + VOP_UNLOCK(rootvnode, 0);
> > > +
> > > p = curthread->td_proc;
> > > FILEDESC_XLOCK(p->p_fd);
> > >
> > > @@ -1496,8 +1498,6 @@
> > >
> > > FILEDESC_XUNLOCK(p->p_fd);
> > >
> > > - VOP_UNLOCK(rootvnode, 0);
> > > -
> > > EVENTHANDLER_INVOKE(mountroot);
> > > }
> > >
> >
> > Still no luck, I now get a LOR that is similar to LOR 281 right after
booting:
> >
> > lock order reversal:
> > 1st 0xffffff0002c2c7f8 ufs (ufs) @ /usr/src/sys/kern/vfs_subr.c:2083
> > 2nd 0xffffff0002b2a248 filedesc structure (filedesc structure) @
> > /usr/src/sys/kern/vfs_syscalls.c:3776
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> > _witness_debugger() at _witness_debugger+0x49
> > witness_checkorder() at witness_checkorder+0x7ea
> > _sx_slock() at _sx_slock+0x44
> > kern_mkdirat() at kern_mkdirat+0x201
> > syscall() at syscall+0x1af
> > Xfast_syscall() at Xfast_syscall+0xe1
> > --- syscall (136, FreeBSD ELF64, mkdir), rip = 0x800729dac, rsp =
> > 0x7fffffffec88, rbp = 0x7fffffffef66 ---
>
> Remove the FILEDESC_SLOCK()/FILEDESC_SUNLOCK() calls from kern_mkdirat().
Several other system calls have the same LOR and need the same fix. I've
consolidated all the fixes so far into
http://www.FreeBSD.org/~jhb/patches/vnode_filedesc.patch
--
John Baldwin
More information about the freebsd-fs
mailing list