From nobody Tue Jun 29 00:23:21 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 7445811D7107 for ; Tue, 29 Jun 2021 00:23:24 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-to1can01on0603.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5d::603]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GDQDq3HvVz4qdb for ; Tue, 29 Jun 2021 00:23:22 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SeNPsjA+kNTTb8txGQvTZbRcBlxIbK6pRu8cr9KYjsLSodyv2wTFgZZai2bQPDIbl5SMKONgKLHMye3s8Qok8+MkkLHZ+IuuszYbsACb4KtrsyGLENhknA5k7B4zLp/OlQT8Y3tekXzP1m+1qaer2zWiid7Uz9XPwZI3i7Hf5kZEDv9boD4gJSxbzHip0D5EBx71a4SiXV4GOYONBhE5HFmxmXArlApY8ef3i4gdIBOiQNxSYFC8L6hRVt1qtla6am7nOaIU5ivqM7n6cKqp4cs2Ur1uOr9qBg1Z3hJtMjfRhPRDMcVZfTyh+QE02IohVOuPa+SkgYUAJry/7iqOfQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DUg8VRzQLTXgOkRtX6FbcdsqyXa+Pd9m3ZKzE5YTjLg=; b=crl541/ilky8MUqszCQ1++bfjgt8P9wX7UcaO2dUcIVvQfgAjVBNeaDgrLje6DYbd9YXM1VGEJAWexnNCrUU6DI2RJXNpDBm2KrwsqcClh7Kiv2Jl2sfYLIxh8Kfb45O6+uiSCY7YVdvssDrJ2qgJRKTDDYIY8lFbZZolsbRaeP3ggcGBlIKXXiU16JGlN1EOabFuDzDRTvq1VOAD+NKJl9ZJPU66bz19sc0C1Ht0n9BzZwC+dDon8psfWSXjZwTYCMvqqDWrLqWFtr/+2s31cZS4TbpzxSdVnLnPpTigPAo9AFYXebd56DzllFGOKy3yeYjXuCmajpGL6CY3Ug2uQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DUg8VRzQLTXgOkRtX6FbcdsqyXa+Pd9m3ZKzE5YTjLg=; b=lsrVjYU7BTFqru17rC4p8BMO9bC9ERL+op+s2txYDo0UNxoBx1T+UKRJ4YwMdneuw443u14QIKkOLTKhQLMp5ODNZBRXyaZFUrQ5ZTFFBbSz4J+wnRKAUfwJR8DPcXWKADoEQWsSigbgK1NP+R16zSTQC0m8dUZzA605N0i8cpaWWlrIMZtrevOWUXApn2jsD8PHrGG5njjjHn0QnXtmfn25J8ZNTiDtPKJIkl9QHEQc3hjVWmaSXwwBXZo2SOkZKxyWOL0FGFSFUbPi/Sr7o5wQ3s2+iQTLxAT2F7wy8xwPV+L70DMvnZhwjQox05v62feJGtquwO4SacyPfeViNg== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by QB1PR01MB3988.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:2e::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4264.19; Tue, 29 Jun 2021 00:23:21 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::583:528b:dbac:37bf]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::583:528b:dbac:37bf%4]) with mapi id 15.20.4264.026; Tue, 29 Jun 2021 00:23:21 +0000 From: Rick Macklem To: freebsd-net Subject: RFC: NFS trunking (multiple TCP connections for a mount Thread-Topic: RFC: NFS trunking (multiple TCP connections for a mount Thread-Index: AQHXbHjbAJ3BuaRhHUuq5k+hERhS3w== Date: Tue, 29 Jun 2021 00:23:21 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 8896fa08-8b72-4fad-8001-08d93a941a06 x-ms-traffictypediagnostic: QB1PR01MB3988: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6790; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 0IeJOBd9171w7vSSoULG5TN2Xx9d2CUCp/493eKIYbRq5PTeseqs6fHkJfOP8pQ5o6beSILAcmfJFExcODMPykOQbYpufL4R2OUjZpkP2jadWmAQiPOBEjWY87c4upufJBTLX2oh/pTIyW7VMvdiESm7q7+DSa72uUJI+XULgwVmmaPDn3Wmi4rXXIl30jr3m9aCNjpF5B/e4H1q6OX49zHHYVHD1+2EFLOHG2XKSHgI5N5v8ZiKAtrgji5zqGiF0/gnqwi5Ff7wFNnAfuCxB8r8R/Re8Ta7sqQM9hlUi9umlhbhBfh18kWfkto/fUDOCQv7d05z3sax9ddASlBgwbHaFo0aNqnMDuzDzQIRa3ht6YV8mSeimH4LHWMyt7gQdhW6XzrrsvpFXaPC+K6+c8qLRzOpt8rLnIr6A+xaXDfDfdsk1NCxkyov3snnHeUIW7a+ZzA9GCcASL0Vqo5o3lZhLTmPkRlQYUxsS3Hxla+G8EBWrG50agOL3BEpEVU1WF80HgzgLyo37x4h2VwBhBaPUvfMzVroBk1nIa2MoI6PdXK2Qe6IUVir83SmvbDVjaosGgsv990aFkzx2RRCB6gyHnpKVA+7ujvzcnaPZjd5TNTSgb/jvU+H1j85pexw46QrET6BbI3whZQuUgAF1w== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(136003)(396003)(376002)(366004)(346002)(39860400002)(5660300002)(86362001)(186003)(71200400001)(6506007)(33656002)(9686003)(83380400001)(6916009)(2906002)(55016002)(66556008)(8936002)(66446008)(8676002)(52536014)(316002)(122000001)(478600001)(7696005)(64756008)(786003)(66476007)(76116006)(38100700002)(66946007)(91956017);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?YBbNtz7Zg7fjaU3frahfIkhybE3RM68QBpxxuQTJVkg6P3EGGlHKZ+svFi?= =?iso-8859-1?Q?zqbQhGoonJ0ooPj8T3fq/o4CRFJ3f1FPDOETq2L8gXRML5qO6ZlUuO2flj?= =?iso-8859-1?Q?Z7zxJ562IK5UAJKO/jePAWrj91R/VrrGmuorCv6yKeSVjCv1LM2pQ8vUvw?= =?iso-8859-1?Q?0jkRJLjyeto0CiZswIGx7D9GhrNnkYDk4MD6OuiMDZvg/Kbuj82jER8eTD?= =?iso-8859-1?Q?nRUaBGYSW+OWdJhrfpgWuc5YTLciKGPsZHEK+AqIjx9GIsH3HkUSzaEX31?= =?iso-8859-1?Q?zeQlrsVgZULdoLdgOQj7wXF9s1lggAZdU/LiuxYpTZnW3ENdtxeh6eBXwm?= =?iso-8859-1?Q?D4JsuwONVL8v1WN9obszdEaqejMjuYgKEpgYSrruGyXv7t+8aA8EX4tIuE?= =?iso-8859-1?Q?8toXBTInajM8l1mDWr2WuUeye9X3OhKkpd9gFJ8iMNTE6P88pIbmaZyz0n?= =?iso-8859-1?Q?L7ME8g2c/DR+2BtaZEYV5IgiqI44D+HuO58ZDgohcFOtNiOv47L7KlwXt8?= =?iso-8859-1?Q?kXS8t8hBRquEFvvwEd3ZEFKjOyD7Nh1zwnhiK2s2AOXcXqUfhPgvzg0ot3?= =?iso-8859-1?Q?toYxqf9m6nv8jdsmwPbY0BirWjeTLtBh20DduV12PLoZQzT4p3Ctvup3ix?= =?iso-8859-1?Q?ZLBn9t9KCnFwrmUtEF9o1hwa4cciA3Qp9enyl4rpq8/SoIbFz/41yVYTC7?= =?iso-8859-1?Q?NtU+WILx4In4p2O/dTLoCDeQYbx7jQstRIn6zoCpf2WMQzuRazjocN42Z6?= =?iso-8859-1?Q?mEn8FkQG0gA3dLmUG9ZuKvGXcLGkqryOgULioT8/dj6nVL8lj32G8d3OHG?= =?iso-8859-1?Q?s4ZHfuOHgbpfo47R98xOHodC4RtUUewX5FKI3HYBbW6y4Pkg9BTPLw0R0B?= =?iso-8859-1?Q?2x9Owd4lfnVYP/6WB6sf6QCVvjRWh8pb6ZrmdAkKEZy1a0PuEoSG7IXCSn?= =?iso-8859-1?Q?Xvbol/CtCGmmlDzeAogE61AnDm/qzc3ToqJxbsX91OmCqxlM7rNtcwNKcw?= =?iso-8859-1?Q?TUnLRgmbozJ3ZzUOl88qnqABgTHUCA0qdWSNbMH9j7yWMFN8wA94mZKswo?= =?iso-8859-1?Q?iDr04W3ooyHzm+fL83X88BroqlIPWqUvYcr6g5VJFKXgeJBvKB/6aQE5jA?= =?iso-8859-1?Q?WPFaIJe4Raoj4W2GLoGINGzegVsIuKkNhzgxRSpUUyMDxoy9wUtEL+h+Sy?= =?iso-8859-1?Q?dhZHIMuB+biR9UOMOkp6Pi6qVV1bZKdFnSjao4xdoADpvGLBnfW1/90WjW?= =?iso-8859-1?Q?d3SCggtxgabu5Bd+YXEswhnc5Uz6waXJTQMS4hoI9pyjVgzwa/1+Awe9ba?= =?iso-8859-1?Q?3gqk8CAnjkqm90xIxtWuCdYKPvuUDYntLAophOy9+YP+ZlFT5KoWm+QVxf?= =?iso-8859-1?Q?bFpJbcGVVYvNvNZuOier4dIs+CnLRjRwZ1zrVMp2e2BbqJYR0Nynk=3D?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 8896fa08-8b72-4fad-8001-08d93a941a06 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jun 2021 00:23:21.4910 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: qHjouwmki3q4fjAg5xwcBVnTlrVvTI+lmZLpE3S4P9gshDBUXCRgDm7gG1QHKYezjIH0BNqrr0tWvWu3GHSFCw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB3988 X-Rspamd-Queue-Id: 4GDQDq3HvVz4qdb X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=uoguelph.ca header.s=selector1 header.b=lsrVjYU7; arc=pass (microsoft.com:s=arcselector9901:i=1); dmarc=pass (policy=none) header.from=uoguelph.ca; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 2a01:111:f400:fe5d::603 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-5.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a01:111:f400:fe5d::603:from]; R_DKIM_ALLOW(-0.20)[uoguelph.ca:s=selector1]; FREEFALL_USER(0.00)[rmacklem]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a01:111:f400::/48]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a01:111:f400:fe5d::603:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[uoguelph.ca:+]; DMARC_POLICY_ALLOW(-0.50)[uoguelph.ca,none]; DWL_DNSWL_LOW(-1.00)[uoguelph.ca:dkim]; NEURAL_HAM_SHORT(-0.98)[-0.984]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; MAILMAN_DEST(0.00)[freebsd-net] X-ThisMailContainsUnwantedMimeParts: N The Linux NFS client now has a mount option "nconnect",=0A= which specifies that multiple TCP connections be created=0A= for an NFS mount, where RPCs are done on the connections,=0A= in a round robin fashion. (Alternating between the two TCP=0A= connections for the case of nconnect=3D2.)=0A= =0A= The Linux man page says:=0A= nconnect=3Dn=0A= When using a connection oriented protocol such as TCP, it=0A= may sometimes be advantageous to set up multiple=0A= connections between the client and server. For instance,=0A= if your clients and/or servers are equipped with multiple=0A= network interface cards (NICs), using multiple connections=0A= to spread the load may improve overall performance. In=0A= such cases, the nconnect option allows the user to specify=0A= the number of connections that should be established=0A= between the client and server up to a limit of 16.=0A= =0A= I don't understand how multiple TCP connections to the same=0A= server IP address will distribute the load across multiple network=0A= interfaces?=0A= I thought that lagg would have handled this?=0A= =0A= I could easily implement this, but I only have low end hardware=0A= to test on, so I doubt that I will see any performance improvement.=0A= =0A= However, I do think that having two TCP connections, where the=0A= RPCs involving large RPC messages (Read/Readdir/Write) are sent=0A= on one TCP connection and the RPCs that use small RPC messages=0A= (Lookup/Access/Getattr,...) are sent on the other one.=0A= --> This would avoid the frequent small RPCs from getting "logjamed"=0A= behind a bunch of large 1Mbyte Read replies, for example.=0A= =0A= So, what do you think?=0A= - Implement "nconnect" with round robin RPC assignment.=0A= or=0A= - Implement two TCP connections where large RPCs are done=0A= on one and small RPCs on the other.=0A= or=0A= ???=0A= =0A= I will note I see downsides to doing multiple TCP connections/mount.=0A= 1 - Uses up more IP port#s.=0A= 2 - When an NFS server gets overloaded, it will stop receiving RPC requests= .=0A= This will eventually apply backpressure through TCP to the client to s= low=0A= down RPC requests. Having multiple TCP connections would reduce this= =0A= backpressure effect.=0A= --> To be honest, I suspect the slowdown in RPC replies caused by an= =0A= overloaded server, is more effective feedback to the NFS client= =0A= than TCP backpressure, but I am not sure.=0A= =0A= Comments? rick=0A= =0A= =0A=