From nobody Wed Jan 29 06:36:07 2025 X-Original-To: current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YjXVL3p7pz5mCKN for ; Wed, 29 Jan 2025 06:36:10 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YjXVL3JPQz3Txd; Wed, 29 Jan 2025 06:36:10 +0000 (UTC) (envelope-from glebius@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1738132570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7cIkCJEMAULHgcetrr8xTpCQ4r9Or0OM8UpO3OabObc=; b=F15kpTQl3tfj8KFui+ZXX+WDyFpNCvxP4FEy26m23sb0EfgUQ/6hxqgPN1fMS4Szb6NLh9 mxOlpAlqv+zQCwyoPFkN2hUIFe4G2hFyLt55kdjK4+NzOa1vcIwZ/ulfNKLlf6U3Onh35i nXu+jEFhE8WUOjsRSezai7jSnB1TH3v7qOZZX/Kt8cdk+V4jLMSWCcBI6gvDrzBaCW3tQu 1FGt2qG/u6RSzFeNxbq34RY1p3AgRj0BbSsDljiiS5lmAL6v+RIbvkf3z8z0fh4cqpfeZa ulQMvhiDKhrmiXWbJ7sRC/ML8/h+BpQ5KQLfqCQv4uJofe2Udp8rSkTy6VaB1g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1738132570; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7cIkCJEMAULHgcetrr8xTpCQ4r9Or0OM8UpO3OabObc=; b=JAXNlsP5qMQioysdvEUiA7i2EF1pReur9M7TIlwFWr11FM/9sQaTj2SXlfUlfKN5GI8Am/ rVlYkrxikZFfxokIVI4r8nQNYmhA/ju33ZBlE5w4BYEfyGyyxr+s+rkujufIrYx3qo0NSD NehEQTi70Y8OI1yGPaEyT7lNaWY4IULpqZ9V+bojYEFjl8k+LqH/pQRRRA5R7Zyov8qs7b A9jffgLPc1zVKP1+syhDcLoK2Plo8oUstL5Cp+OOUuX7FEvAnc/Ue8H6yq/oCHZxl7/SWP 9NU7mXMuqaWBBluEtS9FfFmvh8raMYn+wOmmMxdIYywzuwdgXWF0IN4d9RFd1w== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1738132570; a=rsa-sha256; cv=none; b=aG6o64LwulOtIhIMaLdTfWHKk3FvdvBRhk//TYYoZ0wETvVVxcxXTQkJPzp7rwTE/m/1Sp Ij1H3/2hcWeMA7KRD4w7J1qWkgnaqqugFf9ENjTf1RFMunwuq1F7LT78hYnNZZNkJcABQu 37K+SwoIjqfmcxkhpiO/UTUR9HPtJZv/PZd4U7qWb0u9KPYSh+xW20hN56w5MNSuUAGVmT BcuwIZo0wpWLj9RQV+0AAwjw3hDgwJNgtVacQXgEMv4HjN84sMgKc2lIcvdLk3Wlm9+Fpm rPbnd/M1kdZV1ME796uD+iw8NeZPsN6X1h45Eji8EoqJGbayIlx4+CiM/bW0cA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none Received: from cell.glebi.us (glebi.us [162.251.186.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: glebius) by smtp.freebsd.org (Postfix) with ESMTPSA id 4YjXVK6T2lz7gW; Wed, 29 Jan 2025 06:36:09 +0000 (UTC) (envelope-from glebius@freebsd.org) Date: Tue, 28 Jan 2025 22:36:07 -0800 From: Gleb Smirnoff To: Rick Macklem Cc: current@freebsd.org, rmacklem@freebsd.org Subject: Re: HEADS UP: NFS changes coming into CURRENT early February Message-ID: References: List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: On Tue, Jan 28, 2025 at 09:14:00PM -0800, Gleb Smirnoff wrote: T> Second, with the patch the M_RPC leak count for me is 2. And I found that these T> two items are basically is a clnt_vc that belongs to a closed connection: T> T> fffff80029614a80 tcp4 0 0 10.6.6.9.772 10.6.6.9.2049 CLOSED T> T> There is no connection peer connection, as the server received a timeout trying T> to send. But rpc.tlsclntd doesn't try to send anything on the socket, it just T> keeps it select(2) fd set and doesn't garbage collect. T> T> So it is a bigger resource leak than just two pieces of M_RPC. I don't think T> this is related to my changes. Here is what is going on here: - TCP connection is teared down and tcp_close() calls soisdisconnected() - soisdisconnected() calls clnt_vc_soupcall() to notify of error condition - clnt_vc_soupcall() tries soreceive() and gets so->so_error. - clnt_vc_soupcall() sets the client to error state. It doesn't wakeup anything cause there were no running RPC requests. It can't report back to clnt_rc that connection is dead. It doesn't mark itself for the clnt_vc_dotlsupcall() processing. So we end up with: (kgdb) p $tp->t_state $25 = 0 /* TCPS_CLOSED */ (kgdb) p/x $tp->t_inpcb.inp_flags & 0x04000000 /* INP_DROPPED */ $27 = 0x4000000 (kgdb) p/x $tp->t_inpcb.inp_socket->so_state $28 = 0x2000 /* SS_ISDISCONNECTED */ (kgdb) p/x $tp->t_inpcb.inp_socket->so_count $35 = 0x2 (kgdb) p/x $ct->ct_rcvstate $29 = 0x41 /* RPCRCVSTATE_UPCALLTHREAD | RPCRCVSTATE_NORMAL */ (kgdb) p $ct->ct_error $30 = {re_status = RPC_CANTRECV, ru = {RE_errno = 13, RE_why = RPCSEC_GSS_CREDPROBLEM, RE_vers = {low = 13, high = 0}, RE_lb = {s1 = 13, s2 = 0}}} (kgdb) p $ct->ct_pending $31 = {tqh_first = 0x0, tqh_last = 0xfffff80002838ea8} Note: In my case so->so_error was EACCESS, cause I used ipfw(4) rule to tear down connection, for normal TCP timeout should be ETIMEDOUT or ECONNRESET if remote has reset. That's why $ct->ct_error.ru.RE_errno == 13. So we need some mechanism for clnt_vc_soupcall() to report to upper clnt_rc that we are dead and ready to be garbage collected via CLNT_CLOSE() and then CLNT_RELEASE(). Once clnt_vc_destroy() is called the daemon will be notified that the TLS socket can be closed by the daemon, bringing so_count to 1 and then final sorele() will bring it to 0 and free. -- Gleb Smirnoff