From nobody Wed Nov 02 18:46:16 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N2bSn09dsz4gdMt for ; Wed, 2 Nov 2022 18:46:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4N2bSm4VpMz4810 for ; Wed, 2 Nov 2022 18:46:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4N2bSm3Hpbz16Zq for ; Wed, 2 Nov 2022 18:46:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2A2IkGrJ001927 for ; Wed, 2 Nov 2022 18:46:16 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2A2IkGD7001926 for bugs@FreeBSD.org; Wed, 2 Nov 2022 18:46:16 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 267533] Zfs Multi-Modifier Protection trigger an activity check even if it is disabled Date: Wed, 02 Nov 2022 18:46:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mathieu.schmitt57@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1667414776; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=oZW2jw2TzKKSq0oBB8Y3nK5rHj/bj7RzkQGL40sEt7s=; b=JnrtWlmNYI7MOiTVVKGYPdxDAlwFECzLWsgEP+541xrpBDy86Z03+5qEpNiE1s/AYC/zgl WDkyC4Mi4yrFHhTdG1KxvsUlQ/E16ybB2K4GHW2CQYHeg9SOp8L1P1fJtKhzXVD+x9XMuW 9YVwgsNt/7XlNzbLVHh1nrlel2RPuE2AIoVHso1JLrhzeO/0DAYopBp/fXA8J7VttZ9vnI lV83wtU+G4/im+5OTWOk0mUWdj2+P7goPOkFyX8SM4uGJprx/P2pyJDjiwbXDD0G/ZsPuH bvqcRVx6diVGIieCCR5MhYNFaVn5uyg/WIl5Hc+y5PElKUA81myZpnkuMvj7jw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1667414776; a=rsa-sha256; cv=none; b=pBRv2WY0QXVdD+16O5qPk/kVJO0rMNUk0mv9w3fHk/c2OIbEIGZnlkMv/N21bSr5nXB2Wl F7VCGCqmf3DBbhWbTQ8P2zYdiRqALEGE3BnbeCvjmb66vFyXQtSugHM+XSn4+rUIIMQE+Z LdowfBrQh09V3Rpni+6Nhw2j/bEDwFO+pPXx7hvWqxIsGgEtlq40CtcbSKoha4rXvjph7e gDH19zY/c1ac4JfD4QAFoyZrdJaSE+fcr3gIhjKZMa7kxiDoPqUjRPA3likRvcFpixs98W i21lAWzjkM9gO5j+jW1s/RmaH62WnPWenhyxfhmYU2nmpsiAsTR3i+ltOqUzDw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D267533 Bug ID: 267533 Summary: Zfs Multi-Modifier Protection trigger an activity check even if it is disabled Product: Base System Version: 12.3-RELEASE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: mathieu.schmitt57@gmail.com Created attachment 237826 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D237826&action= =3Dedit git patch for spa.c We run storage clusters in production. These clusters are composed of a pri= mary server and a secondary server. We have launched a campaign to upgrade these clusters from Freebsd 12.1 to 12.3 (amd64). The process of upgrading a whole cluster works as follow: * ensure zpools are live on primary server (12.1) * upgrade secondary server from 12.1 to 12.3 * import zpools on secondary server * upgrade primary server from 12.1 to 12.3 The issue happen when we import zpools on secondary server which has been freshly upgraded to 12.3. Off course, for data integrity reasons, we perform a zpool dry run on 12.3 (secondary server). Dry run exhibit a lot of state: UNAVAIL error on zpools. pool: ssd-00XXXXX id: 10784552113000492957 state: UNAVAIL status: The pool is currently imported by another system. action: The pool must be exported from XXXXX.XXXXXXX (hostid=3D9f7c3a3d) before it can be safely imported. see: http://illumos.org/msg/ZFS-8000-EY config: ssd-00XXXX UNAVAIL currently in use mirror-0 ONLINE da20 ONLINE da1 ONLINE Some logs during a zpool import: spa_load($import, config untrusted): using uberblock with txg=3D3144 pool last imported on non-MMP aware host using import_delay=3D20000000000 multihost_interval=3D1000000000 import_intervals=3D20 disk vdev '/dev/da1': best uberblock found for spa $import. txg 3144 [...] multihost activity detected txg 3144 ub_txg 3145 timestamp 1666367872 ub_timestamp 1666367903 mmp_config 0 ub_mmp_config 0 [...] We have a host with non-MMP and right after, we perform an activity checks. We believe that's the root cause is MMP (Multi-Modifier Protection). According to man zpool, (see: https://www.freebsd.org/cgi/man.cgi?query=3Dzpool&apropos=3D0&sektion=3D8&m= anpath=3DFreeBSD+12.3-RELEASE&arch=3Ddefault&format=3Dhtml), multihost is set to off by default, but the log above show the opposite. In the zfs service pool allocator (spa.c), some checks are performed in the function spa_activity_check_required. One of it looks if MMP is disabled (https://cgit.freebsd.org/src/tree/sys/cddl/contrib/opensolaris/uts/common/= fs/zfs/spa.c?h=3Dreleng/12.3#n2520), and if so, the activity checks is not performed. So in our case, the function spa_ld_select_uberblock that initiate this che= ck will indeed return with an error (EREMOTEIO): it puts the vdev in error w= ith spa_vdev_err(rvd, VDEV_AUX_ACTIVE, EREMOTEIO). In order to get rid of this error during (our) upgrade from 12.1 to 12.3, we thing that this patch can fix the issue: diff --git a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c index 1b452501819..fa0af4636a6 100644 --- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c +++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c @@ -2552,7 +2552,15 @@ spa_activity_check_required(spa_t *spa, uberblock_t = *ub, nvlist_t *label, if (state !=3D POOL_STATE_ACTIVE) return (B_FALSE); - return (B_TRUE); + /* + * Skip the activity check when a pool is imported on a + * non-aware MMP host and the pool was previously imported + * with force option. + */ + if (!MMP_VALID(ub)) + return (B_FALSE); + + return (B_TRUE); } The definition of the uberblock (https://cgit.freebsd.org/src/tree/sys/cddl/contrib/opensolaris/uts/common/= fs/zfs/sys/uberblock_impl.h?h=3Dreleng/12.1) from 12.1 to 12.3 has changed, so when a 12.3 read uberblock initiated by 1= 2.1, some macro are not defined especially the MMP_MAGIC. This MMP_MAGIC is defined (macro) in 12.3 to 0xa11cea11, but is not defined= at all in 12.1 (so the value is set to 0000000000000000), as we can see in the= log below: [-> SLAVE root@XXXX.b 03:59 PM:~]#zdb -luuuu /dev/da14 | grep mmp mmp_magic =3D 0000000000000000 mmp_magic =3D 0000000000000000 [...] This var is (probably) not well initialised, so the check which intent to determine if MMP is disabled, failed. That's why we add the check of MMP_VALID (defined in https://cgit.freebsd.org/src/tree/sys/cddl/contrib/opensolaris/uts/common/f= s/zfs/sys/uberblock_impl.h?h=3Dreleng/12.3#n53) at the end of the sub spa_activity_check_required. Math. --=20 You are receiving this mail because: You are the assignee for the bug.=