Re: git: 8ef0c11e7ce7 - main - nfsclient: upgrade vnode lock in VOP_OPEN()/VOP_CLOSE() if we need to flush buffers

On Tue, Nov 23, 2021 at 07:09:08PM +1100, Peter Jeremy wrote:
> On 2021-Nov-16 17:14:55 +0000, Konstantin Belousov <kib@FreeBSD.org> wrote:
> >    nfsclient: upgrade vnode lock in VOP_OPEN()/VOP_CLOSE() if we need to flush buffers
> >    
> >    VOP_FSYNC() asserts that the vnode is exclusively locked for NFS.
> >    If we try to execute file with recently modified content, the assert is
> >    triggered.
> 
> I have a diskless arm64 system configured with swap over NFS and I'm
> now consistently getting a panic during shutdown.  I haven't
> specificially confirmed that it's this commit but the content is
> suggestive.
> 
> panic: upgrade of unlocked lock (lockmgr) nfs @ /usr/src/sys/fs/nfsclient/nfs_clvnops.c:855
> cpuid = 3
> time = 1637166551
> KDB: stack backtrace:
> db_trace_self() at db_trace_self
> db_trace_self_wrapper() at db_trace_self_wrapper+0x30
> vpanic() at vpanic+0x178
> panic() at panic+0x44
> witness_upgrade() at witness_upgrade+0x104
> lockmgr_upgrade() at lockmgr_upgrade+0x164
> nfs_lock() at nfs_lock+0x2c
> vop_sigdefer() at vop_sigdefer+0x30
> _vn_lock() at _vn_lock+0x54
> nfs_close() at nfs_close+0xc8
> vop_sigdefer() at vop_sigdefer+0x30
> VOP_CLOSE_APV() at VOP_CLOSE_APV+0x2c
> swapdev_close() at swapdev_close+0x3c
> swapoff_one() at swapoff_one+0x598
> sys_swapoff() at sys_swapoff+0x12c
> do_el0_sync() at do_el0_sync+0x498
> handle_el0_sync() at handle_el0_sync+0x90
> --- exception, esr 0x56000000
> 
> I presume this isn't intended.  Can you suggest where I should start
> looking for the problem?

Try this please.  It might be also useful to enable DEBUG_VFS_LOCKS in your
kernel config, to catch all related issues once.

commit 815eb81d8bf24d62a4686673d83b7bf8ec9f7d90
Author: Konstantin Belousov <kib@FreeBSD.org>
Date:   Tue Nov 23 11:16:53 2021 +0200

    swap pager: lock vnode around VOP_CLOSE()
    
    Reported by:    peterj
    MFC after:      1 week

diff --git a/sys/vm/swap_pager.c b/sys/vm/swap_pager.c
index 9bc506c9b6b8..395c0fb72639 100644
--- a/sys/vm/swap_pager.c
+++ b/sys/vm/swap_pager.c
@@ -3057,9 +3057,9 @@ swapdev_strategy(struct buf *bp, struct swdevt *sp)
 static void
 swapdev_close(struct thread *td, struct swdevt *sp)
 {
-
+	vn_lock(sp->sw_vp, LK_EXCLUSIVE | LK_RETRY);
 	VOP_CLOSE(sp->sw_vp, FREAD | FWRITE, td->td_ucred, td);
-	vrele(sp->sw_vp);
+	vput(sp->sw_vp);
 }
 
 static int