From nobody Tue Jun 29 20:22:31 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BA4DDDBAE7E for ; Tue, 29 Jun 2021 20:22:33 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-qb1can01on0604.outbound.protection.outlook.com [IPv6:2a01:111:f400:fe5c::604]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4GDwrT30X2z3sg1 for ; Tue, 29 Jun 2021 20:22:32 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HXe0d/rCvdYwFNLsbeRDkMQbRT/YgUZJW5U6YWlbFYveZ9BSucCEQKniJDjLmsNF5hmE1V7QsSPHTQlhqYhfJBcOL+twmJlP4CPQw9FI3E8zQzqdxHC5KYB+10lGuR3px90J3Ikej/uUEUEijDLTMIfdfDHYQ5lk2taaUFnvYZogKHNyko1Dj0MADJ1IFWDIH+A4eIPhW6sIuJDR1OBLSjrXjSl16/CszgZpA+knvG/KAsZwpA2B5kzhbKXxgUHLEjwRikJQFvJzAcRZE1Ayvdw5sAfSa7lEvx26Nw6BjLXKDPMcjOFiOi/E8fYFAIVuajYHu0hWl3Gxq0cEXBohlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NlSPD5013z9K/zHXGe90rEVXtwrzVDSK4MgQd2Lvwyg=; b=aGg3Ul3yuugFMsaneFhIZjKrIQANEcBWWBWUeNeU4/x5WXofbMxvBC+e6bbxxpYQ2OeRdE6PN6wO2M0ts7GVNn7KFPIdnt+UzxDJ44wSTl4xkbMmKN42nAm1uMQoslR7gB91dOq4JTYeXJrI9vBxFYMzHgJ/Ch/nSYsDOq4ZqaOgqmm1EYAqiCYhJXic6sdc6yUHQZmNYyUAdzCckE7wm+UCcqRNaBdRa2yP054QtH43kCj7U0Jz/vIk47ezw8nxbCG7XVJDKvqdv8jVdpLOj4VO10UKetpdNAqF1j3pUjYULdxO0LpWGaVkxDF/K3waRVkd/tOuotbqwQUgSWR4NA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NlSPD5013z9K/zHXGe90rEVXtwrzVDSK4MgQd2Lvwyg=; b=hk3TUBx71ZWiiQKrWNHaW9xC3HIaZfqvhwYshiaiqVZc/NHNsoF9gQlAIMYFCxUKDlqotkdIII5P9I9SO11LcuSH2hX/pro5JcgiXm3i6wsJwY31aPSxYMEi6rP4Tp/pizAAxRtvEMkrPNChPqskL2pezh8gElvjk7+3R22scLtaZEcmVOX0IkqXlMNTypA5kIy6uDy+nkEzhIQ4RwvBIApSmySXtZUXcxFxFGqV3IG8vn/WDb1Ice7VradIdEoLoRlOzT+KEwsDgzmLPLbDYLISlYjBtOBZkHJA2vX6hHFMg9zLnkhlizSzCwzTm7fDWsuCy1lZTXsBsQeDhJUq3Q== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by YQBPR0101MB4665.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c01:14::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4264.19; Tue, 29 Jun 2021 20:22:31 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::583:528b:dbac:37bf]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::583:528b:dbac:37bf%4]) with mapi id 15.20.4264.026; Tue, 29 Jun 2021 20:22:31 +0000 From: Rick Macklem To: Peter Eriksson CC: freebsd-net Subject: Re: RFC: NFS trunking (multiple TCP connections for a mount Thread-Topic: RFC: NFS trunking (multiple TCP connections for a mount Thread-Index: AQHXbHjbAJ3BuaRhHUuq5k+hERhS36sqtG0AgAC6nLU= Date: Tue, 29 Jun 2021 20:22:31 +0000 Message-ID: References: ,<362300CE-30DA-4552-A3E4-0F3DFE385B2A@lysator.liu.se> In-Reply-To: <362300CE-30DA-4552-A3E4-0F3DFE385B2A@lysator.liu.se> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b3ecb38d-febe-41ae-0933-08d93b3b9f61 x-ms-traffictypediagnostic: YQBPR0101MB4665: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: hLNzHs3Vj1r9gBmt7G184/y62TmZUSuJ9Oc6NJUBEA4QcsTt2Mxl7+r5h2XzkbzZznR6kXlakHdq4JqppVlb9RAfaYLwHVWL4YZfq6zWdLEXCNPzVXW1mxcvPheW5MrL9sHHcgZWZDcqBn5z2qQTsRohniNpezA/5JjmCK/BjHfb2fcekMp/Zmm4uoedOSK9tLSijFf+x/r21Dy3KV2vTXNTsMQ+eHVoG9Uq87T/tjx4B/IUwgGCGj4gfJQMJwVL3Tpi6Z8hIPERXbNghM8GVCi0CsyQpq8Hq7Q6YoPgST//2hJU9caqhbfnv6QKLZDzDpADrGBY3b3a0fcHKysBQ2LwDqaUGrVm1xrUXQhD9f7JlqTkPgjjT+c50w/FfCceEAxsxqzfrythR3XzS4eTFljuK7b+zWurgbL7iKd8MskGpvceVP8WvTAJHRzMXA7pLyBorlv0z8nUsMb0Vmy0Xoq6ZP3KXGfoXJwYNHzXimQxPKiIl/Mrn8nULoT2e86uBHZznHYdQTjqm69soB54n+L2armVxaY2q7hm7QuxuirjUCOkF9Le6vCiVeyLHWv99dY+MsTuYuZMt6PhvRZyWmwv81mp5nL6zc+KFEYOYEZOpIZWMLk470BuRla/CLVoRbdhM66rlj6nyC6LNs+vOg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(396003)(346002)(39860400002)(136003)(366004)(376002)(52536014)(8676002)(66946007)(86362001)(66556008)(478600001)(2906002)(64756008)(66476007)(66446008)(4326008)(6916009)(76116006)(91956017)(55016002)(122000001)(296002)(7696005)(786003)(6506007)(71200400001)(316002)(5660300002)(33656002)(186003)(38100700002)(53546011)(83380400001)(9686003)(8936002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?r49RlVAdS2zUWDOgKr5Ymn4xaufhAwEDdl5SyEDHqMKFujOc1+++V7AE?= =?Windows-1252?Q?1oZi/PsUMJXpqvHke9nZveuqn5miZ9u1ZAjqgS9xTxOSeSL6wR7tp3iL?= =?Windows-1252?Q?kqy3oI8vSgfVjXCIlUsD3zWZqXbfoNdPU659ZH5/m025fzslXZGqmKmR?= =?Windows-1252?Q?hKoNHo/HNxhSlKU2RPycbFewQ27x8tq/QA0vu/uJI4Kz/jBBxP8CcOEJ?= =?Windows-1252?Q?PLRPnu0y8FuqNTeCE6OXikGsKXJiV0HAURxNtUg/PhjGYGetCtXduhCg?= =?Windows-1252?Q?NIhqSJS8NjxSuBOsL2R2QLmqkSpTiNsL1dF+6j++xiA4x53o04rFcGSl?= =?Windows-1252?Q?MtIlqRYs1ZDBLw2zaoFI4l5YAARc0mIEwVWSdLb6XFGmaKfz3IKUecnD?= =?Windows-1252?Q?3CuMisWiEv2iM/p24lyq/pu5Pm1ix+tkYifv0jDRExFB+sonhzW2eH+J?= =?Windows-1252?Q?PunuiLtrfTk2eqFVmr64mUaxWyXLoo06F0ZMTbfNFFkXejx3SJ1VNtvD?= =?Windows-1252?Q?MAxWYKj3TCeDCeWczdxRNkNlcp2p/86heyb7P55FUwMJIAoHq13wWutf?= =?Windows-1252?Q?ig4s7yzVtH50DKr4INiikcyqYwLwEvVKDX5QmP3Ngdo8A4gpEjGyjDz0?= =?Windows-1252?Q?/t7XZMglRJ6j7a1xeHFwef1YsA+5kux3w91oGbTmcsrk1zI0Di/Xkc5q?= =?Windows-1252?Q?lvtbpb1mOJvHVaR58gzG3zMrqCTINceJHvcHrxePkPkIl9vbxZCm0nH0?= =?Windows-1252?Q?NRC0MuAidbCSFlr1laFgJb+uoJD1rwpXM1gLGngF8PqEs5GTJGqcJL7f?= =?Windows-1252?Q?EPWJNFOB/KklqQ1Ekf23ARKgRDhXiKz3PccTItWpMCOsEwzhPcVvV8h6?= =?Windows-1252?Q?Pp6NzRK8QMGK2ha04KzQlOEyKu5bW698I8+o+ZOtzWGJVKA3hk0m3hgx?= =?Windows-1252?Q?Uj7Bn0vI7YcMFbmR3X39Nz1n8pg8SJoBMxFsc20NcRgj1XVbCzvE1iXt?= =?Windows-1252?Q?sxxKYQr5HOrl5qQgQ+eouQTR/qQckpKE6j4ttFfuaLnjTYwcdrmCZtBi?= =?Windows-1252?Q?v4PoDG5NIi7MaO2xtAt4aaj6llXu4r1xcvnsbl1loBUCPrCVTYSRF8vJ?= =?Windows-1252?Q?NvAGkcBAp0z7YeleCz49tNsSrYZZy6f8471bEDrVHFpDdPJ2tKKK6Fnu?= =?Windows-1252?Q?14zcLKfvSQtFfEUfklE0ldqNctV6DRoTF+VKW4d0sqCOvWn6lQks8uwA?= =?Windows-1252?Q?KhHNOE/EdqLRpxXnqMhOthMRuks66S37YUCzt7Jg+75EvgsyVyRUj5bN?= =?Windows-1252?Q?MDpvG8glAl+RgiCuJMbINCyCMEhAN4cVul1cyBpgHxI1PlucsoLJth3/?= =?Windows-1252?Q?nsX5k8ZVS6kGXbpaPOSro/QivfBd9ybmd9XP+YDLe6PEB4ZNWbTpdAjD?= =?Windows-1252?Q?hc5/dy1sGNbmzfHymoa95ldV2F4BG+u9a0pdIqCOyeeIi+ntzEDJdo9u?= =?Windows-1252?Q?yikZoJIJ?= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: b3ecb38d-febe-41ae-0933-08d93b3b9f61 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jun 2021 20:22:31.2162 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: rFU/apBnuwqrDq9t+UCZaWzK/sEFXswUiRp6rDmZXPFsbLKBunEJFOCHlI47tEq5kHAiXVgNaZ2aU+WD815ilQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB4665 X-Rspamd-Queue-Id: 4GDwrT30X2z3sg1 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N Thanks everyone, for your comments. I now see (never used it nor looked at it before) that lagg/lacp pins traffic for a TCP connection to an interface (not infallibly for Peter). I also now learned a little bit w.r.t. multiple queues for fast NICs. Sounds like the "nconnect" option is worth implementing. Thanks for your input, rick ________________________________________ From: Peter Eriksson Sent: Tuesday, June 29, 2021 5:11 AM To: Rick Macklem Cc: freebsd-net Subject: Re: RFC: NFS trunking (multiple TCP connections for a mount CAUTION: This email originated from outside of the University of Guelph. Do= not click links or open attachments unless you recognize the sender and kn= ow the content is safe. If in doubt, forward suspicious emails to IThelp@uo= guelph.ca > I don't understand how multiple TCP connections to the same > server IP address will distribute the load across multiple network > interfaces? > I thought that lagg would have handled this? A lagg typically keeps all data in a TCP stream on a specific lagg member (= depending on how the lagg is set up, unless you select the =93roundrobin=94= option in freebsd - don=92t do that unless you like out-of-order packets= =85) Network equipment with laggs typically hash the IP streams over the lagg me= mbers based on MAC addresses (source&target), IP addresses (source&target) = and port numbers. (We have been diagnosing a fun problem locally where we see packet losses/p= erformance drops over our internal backbone network for certain combination= s of odd/even IP addresses/port numbers when things pass certain SPB =93rou= ters=94 (which typically hash the streams over many =93channels=94 between = routers)=85 Fun fun. :-) I think the multiple NFS TCP streams could make for some nice performance i= mprovements in certain cases. And it would be a more generalisation of havi= ng multiple streams between two hosts - one-or-many over IPv4 and one-or-ma= ny over IPv6 at the same time. Windows SMB has a similar feature. Just avoid the Linux NFS mounting deadlock issue with =93down=94 servers pl= ease :-) - Peter