From nobody Wed Nov 24 15:44:42 2021 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 5326F18AA716 for ; Wed, 24 Nov 2021 15:44:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HzlgZ4y6sz3K8V for ; Wed, 24 Nov 2021 15:44:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8A91B4C01 for ; Wed, 24 Nov 2021 15:44:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 1AOFigGQ051609 for ; Wed, 24 Nov 2021 15:44:42 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 1AOFigIY051608 for bugs@FreeBSD.org; Wed, 24 Nov 2021 15:44:42 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 260011] Unresponsive NFS mount on AWS EFS Date: Wed, 24 Nov 2021 15:44:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rmacklem@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1637768682; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RAFjqFIjilVmHl+i8gPuVhCU4AjvtbUiL4sKIuQVhIQ=; b=IW1w+nGia0oPZT2NMdW8DvoRQ5gQH9NUBLhwl6p9LUBlbT2isNi6FHkrZ9itYnU7jPIyQm NoesFuVY0uMGSDbLmUTCVO4JKiSziemY0YQZbeDQKC1Bk4blXDzQd9YNA3ad8F2O+4EU9F f47pT6hSneV+0xBafJhf3uzypA5lI4ytq3y8gRNstN7RCda+g+wxonduWIt123twkVi0K5 TfXNVQMDoByr4OkvaCwDmVHBxVFOpcPqieFChyX81k4MxpzVG+jbfPaORBQap2n2EFeo/I K/F1IOn83hITkU49MjIRTcV1FUcc0tIkUTQcay6fzRlRX8SjTR2j5UhAn7X3zw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1637768682; a=rsa-sha256; cv=none; b=eXBaDw724S0KmWynOaeWuFy3hSudk3sEWP/I6Cl1j6oy3GWEKsjshOciJvgdHVjsQNxa4e D34GtJDuXo6i+JPhu6P3pZfafR9N+EDmWxXz5uUGSAeEYdRJ+hhnQMUaRpYY2rYK1SuVrO sLvzFj6BZl5k00iW5PCZmSkqFTb3lmr1UvUjiTuRy9Xt66cKsDJLoyxK/bN88pv4tM8ura D6pGM52+rYqjv/0Q5rM4p0+EAAQ7JSEsCV/wH/nwFJ1yds8cciI+24prx/aETWrlbM/Fdb OW0nXhWf/D+u+D+aNVzDOKVR1fStg5DH/JBXTvUQk4hmH8ZG6udF2puokZ56Yw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D260011 --- Comment #1 from Rick Macklem --- Assorted comments: - FreeBSD 13.0 shipped with a bug in the TCP stack which could result in a missed socket receive upcall. --> This normally causes problems for the server, but could cause problems for the client side as well. When you do a "netstat -a" on the hung system, the TCP connection for the mount is shown as ESTABLISHED with Recv-Q non-zero. --> The fix is tp upgrade to stable/13. There is more info on this in PR#254590. - AWS uses a small, fixed number of open_owners, which is why "oneopenown" is needed. If it also uses a small, fixed number of lock_owners (for byte range locking instead of Windows Open locks), then you could run out of these. --> If so, you are SOL (Shit Outa Luck) and my only suggestion would be to try an nfsv3 mount with the "nolockd" mount option. "netstat -E -c" should show you how many lock_owners are allocated at the time of the hang. Other than that, if AWS has now added support for delegations, there could be assorted breakage. Not running the nfscbd(8) daemon should avoid isuuing of delegations, if that is the case. If a soft mount fails a syscall, then the session slot is screwed up and this makes the mount fail in weird ways. "umount -N " is a much better way to deal with hung mounts. As a starting point, posting the output of: ps axHl procstat -kk netstat -a nfsstat -E -c on the client when hung will give us more information. Good luck with it, rick --=20 You are receiving this mail because: You are the assignee for the bug.=