From nobody Wed Nov 24 22:38:53 2021 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 94B7E18B20A5; Wed, 24 Nov 2021 22:38:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HzwsT2bSWz4tMt; Wed, 24 Nov 2021 22:38:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 35A5712522; Wed, 24 Nov 2021 22:38:53 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.16.1/8.16.1) with ESMTP id 1AOMcrTb016837; Wed, 24 Nov 2021 22:38:53 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.16.1/8.16.1/Submit) id 1AOMcrkl016836; Wed, 24 Nov 2021 22:38:53 GMT (envelope-from git) Date: Wed, 24 Nov 2021 22:38:53 GMT Message-Id: <202111242238.1AOMcrkl016836@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Rick Macklem Subject: git: 354988ca3f9d - stable/13 - nfsd: Fix the NFSv4.2 pNFS MDS server for NFSERR_NOSPC via LayoutError List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-src-all@freebsd.org X-BeenThere: dev-commits-src-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: rmacklem X-Git-Repository: src X-Git-Refname: refs/heads/stable/13 X-Git-Reftype: branch X-Git-Commit: 354988ca3f9db745ce0734ef1bda2daba584f723 Auto-Submitted: auto-generated ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1637793533; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=QnEoioRmOq4m0kxphxCSUugkn92UKTAa/PF5vB0KFR0=; b=dZjYEF9B3L/H6tdATj5tU+h0yrf9rMcFOZZ4Q+nnS6ealtEjB3ltlj1Eb9nF629jkOW73m Lzo5aAoaXSab98SBJuI5QqEmpC5OvH9uQp2+ossFEK9z+G/xBksk5iFcjOh7KVQd/eZyf4 uKwGsf5ZNx+Gch1iRInO4nF9gBZc5xJBa8Z548lUFe6yiXNeF8w4qydRVKTo6hOPS0YUVB rVpri5yEn7VfDCDhaDZM5S3Ws6SnS2hxGAW7B7cTRAWdtBogSsWXQhomni5FfQgfNMR2yO R+BYn/x3x4ELVal4v537wZlo9oFTvArmlR+ji/Yab6UJo+z03IGOKPIQpZuUVQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1637793533; a=rsa-sha256; cv=none; b=UbSkQn9Pr7Jr0dWhx25PkBkJ7j7XquwZFi2NnHP+HFfjTj94KbeuCmVTtn1u6muT3RJvE6 gITs7qPuAGJ4qNgn8dujB2dhKXHDyvvDZjtBHjOLmSBw4zov/DJ2/Phn0xYMGHe2U08AMe cugbz69nTj4kG+U5RxTFu1uF4XBc84d78qx+1dyr8zqiwmPJIsDSpBJ/tq85EZNB0UEtFQ UahMrg/dx/o1MscJbr4JBQBR/qrtK9mR4GBTX0lK4rviWM6qYhSMbkMKWfiQrQ9wiStQrq l+k9+VNHiywW7/F2RElJmdojpCMGz1mUo9SYGFQTgJWdFLnJ1c4IdwByfQKHtw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N The branch stable/13 has been updated by rmacklem: URL: https://cgit.FreeBSD.org/src/commit/?id=354988ca3f9db745ce0734ef1bda2daba584f723 commit 354988ca3f9db745ce0734ef1bda2daba584f723 Author: Rick Macklem AuthorDate: 2021-11-08 23:58:00 +0000 Commit: Rick Macklem CommitDate: 2021-11-24 22:35:03 +0000 nfsd: Fix the NFSv4.2 pNFS MDS server for NFSERR_NOSPC via LayoutError If a pNFS server's DS runs out of disk space, it replies NFSERR_NOSPC to the client doing writing. For the Linux client, it then sends a LayoutError RPC to the MDS server to tell it about the error and keeps retrying, doing repeated LayoutGets to the MDS and Write RPCs to the DS. The Linux client is "stuck" until disk space on the DS is free'd up unless a subsequent LayoutGet request is sent a NFSERR_NOSPC reply. The looping problem still occurs for NFSv4.1 mounts, but no fix for this is known at this time. This patch changes the pNFS MDS server to reply to LayoutGet operations with NFSERR_NOSPC once a LayoutError reports the problem, until the DS has available space. This keeps the Linux NFSv4.2 from looping. Found during recent testing because of issues w.r.t. a DS being out of space found during a recent IEFT NFSv4 working group testing event. (cherry picked from commit f8dc06303bac39be53872de7429aa54694b3f86a) --- sys/fs/nfs/nfs_var.h | 2 ++ sys/fs/nfs/nfsrvstate.h | 3 +++ sys/fs/nfsserver/nfs_nfsdport.c | 53 +++++++++++++++++++++++++++++++++++++++ sys/fs/nfsserver/nfs_nfsdserv.c | 4 +++ sys/fs/nfsserver/nfs_nfsdsocket.c | 3 +++ sys/fs/nfsserver/nfs_nfsdstate.c | 53 +++++++++++++++++++++++++++++++++++++-- 6 files changed, 116 insertions(+), 2 deletions(-) diff --git a/sys/fs/nfs/nfs_var.h b/sys/fs/nfs/nfs_var.h index b45143067c46..1f20c1cb76a8 100644 --- a/sys/fs/nfs/nfs_var.h +++ b/sys/fs/nfs/nfs_var.h @@ -170,6 +170,7 @@ int nfsrv_copymr(vnode_t, vnode_t, vnode_t, struct nfsdevice *, int nfsrv_mdscopymr(char *, char *, char *, char *, int *, char *, NFSPROC_T *, struct vnode **, struct vnode **, struct pnfsdsfile **, struct nfsdevice **, struct nfsdevice **); +void nfsrv_marknospc(char *, bool); /* nfs_nfsdserv.c */ int nfsrvd_access(struct nfsrv_descript *, int, @@ -762,6 +763,7 @@ int nfsvno_listxattr(struct vnode *, uint64_t, struct ucred *, struct thread *, void nfsm_trimtrailing(struct nfsrv_descript *, struct mbuf *, char *, int, int); bool nfsrv_checkwrongsec(struct nfsrv_descript *, int, enum vtype); +void nfsrv_checknospc(void); /* nfs_commonkrpc.c */ int newnfs_nmcancelreqs(struct nfsmount *); diff --git a/sys/fs/nfs/nfsrvstate.h b/sys/fs/nfs/nfsrvstate.h index 427d5b132281..9eebeece9727 100644 --- a/sys/fs/nfs/nfsrvstate.h +++ b/sys/fs/nfs/nfsrvstate.h @@ -132,6 +132,7 @@ struct nfslayout { nfsv4stateid_t lay_stateid; nfsquad_t lay_clientid; fhandle_t lay_fh; + char lay_deviceid[NFSX_V4DEVICEID]; fsid_t lay_fsid; uint32_t lay_layoutlen; uint16_t lay_mirrorcnt; @@ -147,6 +148,7 @@ struct nfslayout { #define NFSLAY_RECALL 0x0004 #define NFSLAY_RETURNED 0x0008 #define NFSLAY_CALLB 0x0010 +#define NFSLAY_NOSPC 0x0020 /* * Structure for an NFSv4.1 session. @@ -353,6 +355,7 @@ struct nfsdevice { char *nfsdev_host; fsid_t nfsdev_mdsfsid; uint32_t nfsdev_nextdir; + bool nfsdev_nospc; vnode_t nfsdev_dsdir[0]; }; diff --git a/sys/fs/nfsserver/nfs_nfsdport.c b/sys/fs/nfsserver/nfs_nfsdport.c index f48e57e449c9..bd833f1c375a 100644 --- a/sys/fs/nfsserver/nfs_nfsdport.c +++ b/sys/fs/nfsserver/nfs_nfsdport.c @@ -6733,6 +6733,59 @@ nfsrv_checkwrongsec(struct nfsrv_descript *nd, int nextop, enum vtype vtyp) return (true); } +/* + * Check DSs marked no space. + */ +void +nfsrv_checknospc(void) +{ + struct statfs *tsf; + struct nfsdevice *ds; + struct vnode **dvpp, **tdvpp, *dvp; + char *devid, *tdevid; + int cnt, error = 0, i; + + if (nfsrv_devidcnt <= 0) + return; + dvpp = mallocarray(nfsrv_devidcnt, sizeof(*dvpp), M_TEMP, M_WAITOK); + devid = malloc(nfsrv_devidcnt * NFSX_V4DEVICEID, M_TEMP, M_WAITOK); + tsf = malloc(sizeof(*tsf), M_TEMP, M_WAITOK); + + /* Get an array of the dvps for the DSs. */ + tdvpp = dvpp; + tdevid = devid; + i = 0; + NFSDDSLOCK(); + /* First, search for matches for same file system. */ + TAILQ_FOREACH(ds, &nfsrv_devidhead, nfsdev_list) { + if (ds->nfsdev_nmp != NULL && ds->nfsdev_nospc) { + if (++i > nfsrv_devidcnt) + break; + *tdvpp++ = ds->nfsdev_dvp; + NFSBCOPY(ds->nfsdev_deviceid, tdevid, NFSX_V4DEVICEID); + tdevid += NFSX_V4DEVICEID; + } + } + NFSDDSUNLOCK(); + + /* Do a VFS_STATFS() for each of the DSs and clear no space. */ + cnt = i; + tdvpp = dvpp; + tdevid = devid; + for (i = 0; i < cnt && error == 0; i++) { + dvp = *tdvpp++; + error = VFS_STATFS(dvp->v_mount, tsf); + if (error == 0 && tsf->f_bavail > 0) { + NFSD_DEBUG(1, "nfsrv_checknospc: reset nospc\n"); + nfsrv_marknospc(tdevid, false); + } + tdevid += NFSX_V4DEVICEID; + } + free(tsf, M_TEMP); + free(dvpp, M_TEMP); + free(devid, M_TEMP); +} + extern int (*nfsd_call_nfsd)(struct thread *, struct nfssvc_args *); /* diff --git a/sys/fs/nfsserver/nfs_nfsdserv.c b/sys/fs/nfsserver/nfs_nfsdserv.c index 0e447778d639..f5ff9f8fab83 100644 --- a/sys/fs/nfsserver/nfs_nfsdserv.c +++ b/sys/fs/nfsserver/nfs_nfsdserv.c @@ -5052,6 +5052,10 @@ nfsrvd_layouterror(struct nfsrv_descript *nd, __unused int isdgram, if (stat != NFSERR_ACCES && stat != NFSERR_STALE && stat != NFSERR_NOSPC) nfsrv_delds(devid, curthread); + + /* For NFSERR_NOSPC, mark all deviceids and layouts. */ + if (stat == NFSERR_NOSPC) + nfsrv_marknospc(devid, true); } nfsmout: vput(vp); diff --git a/sys/fs/nfsserver/nfs_nfsdsocket.c b/sys/fs/nfsserver/nfs_nfsdsocket.c index 85771974be2f..002236113068 100644 --- a/sys/fs/nfsserver/nfs_nfsdsocket.c +++ b/sys/fs/nfsserver/nfs_nfsdsocket.c @@ -722,6 +722,9 @@ nfsrvd_compound(struct nfsrv_descript *nd, int isdgram, u_char *tag, p = curthread; + /* Check for and optionally clear the no space flags for DSs. */ + nfsrv_checknospc(); + NFSVNO_EXINIT(&vpnes); NFSVNO_EXINIT(&savevpnes); /* diff --git a/sys/fs/nfsserver/nfs_nfsdstate.c b/sys/fs/nfsserver/nfs_nfsdstate.c index 797b9b0a466e..e6a919093738 100644 --- a/sys/fs/nfsserver/nfs_nfsdstate.c +++ b/sys/fs/nfsserver/nfs_nfsdstate.c @@ -6817,9 +6817,14 @@ nfsrv_layoutget(struct nfsrv_descript *nd, vnode_t vp, struct nfsexstuff *exp, NFSD_DEBUG(1, "ret layout too small\n"); return (NFSERR_TOOSMALL); } - if (*iomode == NFSLAYOUTIOMODE_RW) + if (*iomode == NFSLAYOUTIOMODE_RW) { + if ((lyp->lay_flags & NFSLAY_NOSPC) != 0) { + NFSUNLOCKLAYOUT(lhyp); + NFSD_DEBUG(1, "ret layout nospace\n"); + return (NFSERR_NOSPC); + } lyp->lay_flags |= NFSLAY_RW; - else + } else lyp->lay_flags |= NFSLAY_READ; NFSBCOPY(lyp->lay_xdr, layp, lyp->lay_layoutlen); *layoutlenp = lyp->lay_layoutlen; @@ -6892,6 +6897,7 @@ nfsrv_filelayout(struct nfsrv_descript *nd, int iomode, fhandle_t *fhp, NFSBCOPY(fhp, &lyp->lay_fh, sizeof(*fhp)); lyp->lay_clientid.qval = nd->nd_clientid.qval; lyp->lay_fsid = fs; + NFSBCOPY(devid, lyp->lay_deviceid, NFSX_V4DEVICEID); /* Fill in the xdr for the files layout. */ tl = (uint32_t *)lyp->lay_xdr; @@ -6941,6 +6947,7 @@ nfsrv_flexlayout(struct nfsrv_descript *nd, int iomode, int mirrorcnt, lyp->lay_clientid.qval = nd->nd_clientid.qval; lyp->lay_fsid = fs; lyp->lay_mirrorcnt = mirrorcnt; + NFSBCOPY(devid, lyp->lay_deviceid, NFSX_V4DEVICEID); /* Fill in the xdr for the files layout. */ tl = (uint32_t *)lyp->lay_xdr; @@ -7015,6 +7022,10 @@ nfsrv_flexlayouterr(struct nfsrv_descript *nd, uint32_t *layp, int maxcnt, if (stat != NFSERR_ACCES && stat != NFSERR_STALE && stat != NFSERR_NOSPC) nfsrv_delds(devid, p); + + /* For NFSERR_NOSPC, mark all devids and layouts. */ + if (stat == NFSERR_NOSPC) + nfsrv_marknospc(devid, true); } } } @@ -8773,3 +8784,41 @@ nfsrv_findmirroredds(struct nfsmount *nmp) } return (fndds); } + +/* + * Mark the appropriate devid and all associated layout as "out of space". + */ +void +nfsrv_marknospc(char *devid, bool setit) +{ + struct nfsdevice *ds; + struct nfslayout *lyp; + struct nfslayouthash *lhyp; + int i; + + NFSDDSLOCK(); + TAILQ_FOREACH(ds, &nfsrv_devidhead, nfsdev_list) { + if (NFSBCMP(ds->nfsdev_deviceid, devid, NFSX_V4DEVICEID) == 0) { + NFSD_DEBUG(1, "nfsrv_marknospc: devid %d\n", setit); + ds->nfsdev_nospc = setit; + } + } + NFSDDSUNLOCK(); + + for (i = 0; i < nfsrv_layouthashsize; i++) { + lhyp = &nfslayouthash[i]; + NFSLOCKLAYOUT(lhyp); + TAILQ_FOREACH(lyp, &lhyp->list, lay_list) { + if (NFSBCMP(lyp->lay_deviceid, devid, + NFSX_V4DEVICEID) == 0) { + NFSD_DEBUG(1, "nfsrv_marknospc: layout %d\n", + setit); + if (setit) + lyp->lay_flags |= NFSLAY_NOSPC; + else + lyp->lay_flags &= ~NFSLAY_NOSPC; + } + } + NFSUNLOCKLAYOUT(lhyp); + } +}