From nobody Wed Nov 24 02:19:18 2021 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 9DDB2189B38A; Wed, 24 Nov 2021 02:19:34 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "vtr.rulingia.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HzPpZ1Kcgz4f6J; Wed, 24 Nov 2021 02:19:33 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (2001-44b8-31fc-0d00-c8ab-09b5-49e2-9ade.static.ipv6.internode.on.net [IPv6:2001:44b8:31fc:d00:c8ab:9b5:49e2:9ade]) by vtr.rulingia.com (8.16.1/8.16.1) with ESMTPS id 1AO2JNp0025827 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 24 Nov 2021 13:19:29 +1100 (AEDT) (envelope-from peter@rulingia.com) DKIM-Filter: OpenDKIM Filter v2.10.3 vtr.rulingia.com 1AO2JNp0025827 X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.16.1/8.16.1) with ESMTPS id 1AO2JIlV026739 (version=TLSv1.3 cipher=AEAD-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 24 Nov 2021 13:19:18 +1100 (AEDT) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.16.1/8.16.1/Submit) id 1AO2JIYi026738; Wed, 24 Nov 2021 13:19:18 +1100 (AEDT) (envelope-from peter) Date: Wed, 24 Nov 2021 13:19:18 +1100 From: Peter Jeremy To: Konstantin Belousov Cc: src-committers@freebsd.org, dev-commits-src-all@freebsd.org, dev-commits-src-main@freebsd.org Subject: Re: git: 8ef0c11e7ce7 - main - nfsclient: upgrade vnode lock in VOP_OPEN()/VOP_CLOSE() if we need to flush buffers Message-ID: References: <202111161714.1AGHEtBA084291@gitrepo.freebsd.org> List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="lpqbkyaBU6BTja4R" Content-Disposition: inline In-Reply-To: X-PGP-Key: http://www.rulingia.com/keys/peter.pgp X-Rspamd-Queue-Id: 4HzPpZ1Kcgz4f6J X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N --lpqbkyaBU6BTja4R Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2021-Nov-23 11:19:21 +0200, Konstantin Belousov wr= ote: >On Tue, Nov 23, 2021 at 07:09:08PM +1100, Peter Jeremy wrote: >> On 2021-Nov-16 17:14:55 +0000, Konstantin Belousov wro= te: >> > nfsclient: upgrade vnode lock in VOP_OPEN()/VOP_CLOSE() if we need = to flush buffers >> > =20 >> > VOP_FSYNC() asserts that the vnode is exclusively locked for NFS. >> > If we try to execute file with recently modified content, the asser= t is >> > triggered. >>=20 >> I have a diskless arm64 system configured with swap over NFS and I'm >> now consistently getting a panic during shutdown. I haven't >> specificially confirmed that it's this commit but the content is >> suggestive. >>=20 >> panic: upgrade of unlocked lock (lockmgr) nfs @ /usr/src/sys/fs/nfsclien= t/nfs_clvnops.c:855 >> cpuid =3D 3 >> time =3D 1637166551 >> KDB: stack backtrace: >> db_trace_self() at db_trace_self >> db_trace_self_wrapper() at db_trace_self_wrapper+0x30 >> vpanic() at vpanic+0x178 >> panic() at panic+0x44 >> witness_upgrade() at witness_upgrade+0x104 >> lockmgr_upgrade() at lockmgr_upgrade+0x164 >> nfs_lock() at nfs_lock+0x2c >> vop_sigdefer() at vop_sigdefer+0x30 >> _vn_lock() at _vn_lock+0x54 >> nfs_close() at nfs_close+0xc8 >> vop_sigdefer() at vop_sigdefer+0x30 >> VOP_CLOSE_APV() at VOP_CLOSE_APV+0x2c >> swapdev_close() at swapdev_close+0x3c >> swapoff_one() at swapoff_one+0x598 >> sys_swapoff() at sys_swapoff+0x12c >> do_el0_sync() at do_el0_sync+0x498 >> handle_el0_sync() at handle_el0_sync+0x90 >> --- exception, esr 0x56000000 >>=20 >> I presume this isn't intended. Can you suggest where I should start >> looking for the problem? > >Try this please. It might be also useful to enable DEBUG_VFS_LOCKS in your >kernel config, to catch all related issues once. Thanks for the rapid response. I've found that DEBUG_VFS_LOCKS also requires INVARIANTS. That now catches swapon as well: KDB: stack backtrace: db_trace_self() at db_trace_self db_trace_self_wrapper() at db_trace_self_wrapper+0x30 assert_vop_locked() at assert_vop_locked+0x58 VOP_GETATTR_APV() at VOP_GETATTR_APV+0x4c sys_swapon() at sys_swapon+0x2c0 do_el0_sync() at do_el0_sync+0x4a4 handle_el0_sync() at handle_el0_sync+0x90 --- exception, esr 0x56000000 vnode 0xffffa000065511c0: type VREG usecount 1, writecount 0, refcount 1 seqc users 0 hold count flags () flags (VV_VMSIZEVNLOCK) lock type nfs: UNLOCKED fileid 30984 fsid 0x3a3a00ff01 VOP_GETATTR: 0xffffa000065511c0 is not locked but should be KDB: enter: lock violation [ thread pid 1027 tid 100159 ] Stopped at kdb_enter+0x48: undefined f900c11f I will dig into that further this evening. --=20 Peter Jeremy --lpqbkyaBU6BTja4R Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAmGdoSBfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzRXyg//YhsbNWvS+kiv4L4qtesNi2ZBzqUEx2Hykq9Ww8U3zgauF7rkCurE5WXO 68k8ALwOyDkGPo8oev867rayvHouY6bvErJotlWLdlxHa/aUAHjEBB9ulZ8CBT9C BMZ2TrzKSby5lCmcXpVsGd6diXYjwvkJAXvN/EV8wm9TuRlllWpi5FAzckn+UXzw 5oFktWKPDKB4lMrwZMpTwSU+Cj8/GTSioRmqsnqa7zkT1FjwaNyJ8qJKBBlIpkhK dOe5sjXLRKJaynJqfy12qQPsjeauPPIzFdri7nkH+jINi5e+bBbfhD1UFaBgmlM7 cp9O24nQLi4Sdo0WHfMXyztDiO9+1R3De/Ys3w9whZDySrbqwwIrtzWOdHu0PnpC k8PhHsgPZE0WNqtkUa6BlhjJ9FicfujxztdzywuP+vx2tBFgyIGKQC7uWgoVZ1qR LL5sNOEfEu2XPYeGKBjjaEiJXrgNTCc3KGOjyXfTZEaFgO+llTo865q/30oayhsK IMZRE+A4zyXcw0c/TshxVGYikwWS4CgW7BIrMNwfxLOPasFD0HRaD+taR+Fj1KyA Bn7pvaPAWcUFgoEH2aOGG59j2islQSgYTHa6StxZBBcLPFQNfVEd9+oTLi0yTYV9 sjwuJvfBfjH/TtJ2Z2v46xNHKlDEY+KJrlg+fwK299SZkXS1plI= =Kf6x -----END PGP SIGNATURE----- --lpqbkyaBU6BTja4R--