From nobody Fri Dec 03 23:59:38 2021 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 8C22C18CBCDD for ; Sat, 4 Dec 2021 00:15:30 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660059.outbound.protection.outlook.com [40.107.66.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J5VZp2j5Gz4ldd for ; Sat, 4 Dec 2021 00:15:30 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Y6i2909p22AUBP6EiX2JjU+M6ISjGNQS72brHkA+DKJ5S6oWOh38DB3cFBaJids6ZDkDTRc26GbK3Y0Zq1zEigoRrkozV3b7hnEWzcnmGMr+sXXsi+tVtcGp/gEhIicy5cAF3WAI10IXEWmX5WsCHKbX8CnIvorqAY74ski0E/Bkx8HZ8RRlCtjYOVOgcuEJ2zzTCb2d0nTxipxBxW+mkfxznsVAq3ncxTgCwMnrAspb04c0Wl06+rXIoggXl2Q9l7ZtxMrEvjOvE53e6J7LTGEoNE7LkVnOOhXGXBzyh0KyRseg24Tth4+bKahzBsz8zDUDU/dP77CVUr55KKWgeg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mz9JBlw05p38XbS7No/HOHcxnvEthHYSdqDMxTtkSBE=; b=gekM/zie10pFSSHqvnIi+Ylsve5L53Szo/0sYGD3seIG8kKuZ9SH/Qtudz6qlIYKovZh45L/uyN5xTHwuVNM0xFWXA4T68i2tp0cJwGgnVmQJNd+57n4U/xK+hnLzi9NmfyVgtKl1LjiZrYZsGn8WW/uooa4ceIveU3qYbOefKszEIX/VawB+0hPKSaNKmQWqrTf3rjNAQAv8FZMYf0VvhuocYGHPaTUht7eFh0aV32dfJUGQ1s+WWrPFRxCbAMb0Ojn7W7cITT8XSPQxY9ptVhDedwRr1IF+LWO14oQOgaAQ68vOSb2ferbLHGXKQA8Z8cS745xrUt08Tqw14nrmA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=uoguelph.ca; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mz9JBlw05p38XbS7No/HOHcxnvEthHYSdqDMxTtkSBE=; b=MfV2i6zTjwrhfawFbGH6GysPFKNIUJOM9rg5UNZ7cf1+F62Vbo3Ym2xJlyE+LlH69Ui41cdBh/0cxcSN1PvoZ8VfQhV7+JNcFdKWLZDO1L+VRBKtYkl3aE9G7fRtIVQqmL5iA/Ifb1WY5o7W2mXcOaJTcTvhMtI0lXPVyvPbRBaStwuuKNbTTYYncJ7GWotlD7dmiyr8W65dEILpr29txrDWwngEkcqm+z6tzR8Jc6zjundVAkggQLTozycabdO8uuvEmfOwDG7U5yq++FnCLq6bxwaaM/+dOzP7zx2zxZSaDMX7G2enrup0HM4ebZ0bnRt8NQdSESpJs5QtfahJ1g== Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:19::29) by QB1PR01MB2771.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:c00:36::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4755.11; Fri, 3 Dec 2021 23:59:38 +0000 Received: from YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::e56f:b7a2:3830:5706]) by YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM ([fe80::e56f:b7a2:3830:5706%3]) with mapi id 15.20.4734.028; Fri, 3 Dec 2021 23:59:38 +0000 From: Rick Macklem To: Ryan Stone , John Jasen CC: FreeBSD Net , Navdeep Parhar Subject: Re: Chelsio cards, jumbo frames, memory fragmentation and performance in FreeBSD 13.x? Thread-Topic: Chelsio cards, jumbo frames, memory fragmentation and performance in FreeBSD 13.x? Thread-Index: AQHX6ItisNPRlDj8k02DyDafiCiGCKwhcMoZ Date: Fri, 3 Dec 2021 23:59:38 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: c3d3f36e-0b33-54aa-03b7-cabe6273fbf7 x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 9e2cfd5d-d191-44ef-ddb3-08d9b6b8f6d2 x-ms-traffictypediagnostic: QB1PR01MB2771: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: mFLAS00O3hlq62kSvbWT1x19aqlb6eirZkDvx6l6MHJLG1RMIns3IkJVWAVFKacUeN2fLjhpHt7PBiGh/+2G0gh3UzCO4qGO0Ki5Fz4Ki0h9ogc7zas0+Gsz0NxrP1PCF5rrx1SsYf8k29prPnOBVhARK6/graoqKljSpQ5w3oGU2FyrmbfXDR/k/5hCXhE7a74vHChy6j4KKrUvO1lE8a/ies2deONuHOVHuDv7YGGwM3vVka/kLBKrXolrUI7s7Ku+2q82CcngyKrcU46CeWeBHFCmcq6ouxvsGOMg9Q4xV61BMeBSNY52NC0NYCptl7WDYb1+0ZeTkQ+y4mInSBsXDehnOhiLxxxTJArJZi2zWG1PQVv20RA88P4Zalyz3bn9SbPuaIAvEJ9Zv0d5f8b75XwNhEf07el1kOg8cozCbxzh8nbA56P0ts3qlGIjDBzJV17MYj4L10eASfA5izYE3PLbEfI8TImxEBS20q4igdZB45y4BCb9JS2hWvPJRjPQ0cOVVCCKL4CyWdcv89TTU/tycE3wN0PCDFdmJmfjh3ylbYrkQL2y+FkMIvvUnFmTX395FwH59D74NqeVHKud7A7j3cr2gSxqObzZj1fjbMnJG2izYv3r9qki9GahzzDyi+4kbx6LFpeMyq2KsNO1i0+ruCrdOEX9mRFNUvLJBIvdh8Hg1YOnN8ziFvs3Wj0Pr+YvCdBbwYVJC50CRg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(366004)(91956017)(53546011)(52536014)(316002)(54906003)(66946007)(786003)(83380400001)(33656002)(7696005)(4326008)(122000001)(2906002)(5660300002)(186003)(76116006)(71200400001)(55016003)(508600001)(66556008)(9686003)(8676002)(66446008)(86362001)(66476007)(6506007)(38070700005)(8936002)(64756008)(110136005)(38100700002);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?WjZkijz3U16zdo2Z8VZS8Rf65nG7Z8BEKNAKTkuHP/IR8pFqkDvaY2FwZP?= =?iso-8859-1?Q?jcgJWvnPMyeVSftKyhYF0Zf2Npxbxz0XVmI7iAhMobP9PVsQvJtBX7pfhS?= =?iso-8859-1?Q?N5w55sowi3OQuyNTf4SS2j57uJS1f4HRFjyI1OTEMIwklxXHcsASzodX4W?= =?iso-8859-1?Q?+5gXsJssTHKLMN7+zoYQpPLOIeyEfW6c7aHr5KXXTocyW5ogWfRWoIDeJi?= =?iso-8859-1?Q?11UijkgL0OgI5fJ7NU3tseoexzN90CIDRj7/MfeXyrlqf/r1tgJs4jh6e+?= =?iso-8859-1?Q?XU7yctrqSAEu4ukhqUfXQmacl5s3KOISwl+Bjt2glFu3ZmMp94JORfKDir?= =?iso-8859-1?Q?Ip4OuYXJep2pfNYeftmlUMTcwW8p6FOw92riGrXo2zv4B0uNWgv103inFP?= =?iso-8859-1?Q?9fXMtYfhtPer79XxmDzkQT9mh5QW9F9/Edfywcu+e9gj5t3F2uaXnopUoo?= =?iso-8859-1?Q?9sT73w/N3xZwVaQ5LwtU5+1xTHQbi/Mvj45DSyTbhACyAFY2SuH4vgqDx6?= =?iso-8859-1?Q?5nwdp3SMG/1V5CpjpxqKJt+iZ8TD5TXPE/GiXPWglXfDC4rE4Ek1ptC0l4?= =?iso-8859-1?Q?HVkkcR8k6ZIun9hug0RmLg/tczz9oXTI0vuUjas0DX09R4aenXpyjTN9Zh?= =?iso-8859-1?Q?RUqDL14PtGQttzEXKfLYHsH+2K04yQsoR2/3VPoxPipeG4CJBA4Xfk4Xo0?= =?iso-8859-1?Q?CEzj7jGcCni9CLhwsB4Yo/VzGtOtmfMgArxTNS3X40SuYtjzPgKyMafTzR?= =?iso-8859-1?Q?Okv+HsMwEab7sYCuu3KHTUCNNI7xq8hgEV6sXmprwQlTKeQRGhhhCZj0MM?= =?iso-8859-1?Q?o2A8MJh07tXbQfFASKuoGnXxTvUWQqfBOzwtWr7IIcwVlReMvsV5K1iDBB?= =?iso-8859-1?Q?NJ8J1Il0NYhXuJ/xY/hB8TVtOzDflU7oIyds+QNdfZ3O7rxCJqAP4u/YJP?= =?iso-8859-1?Q?QTU1A+CcQQLy2mh7RMaOhxzQOiRCQ4B4/+IaRWTlwt5YGf1X/vkwceAnax?= =?iso-8859-1?Q?jdhm2lvOcT/tKsIjtBjtFp+Q+LVi7AI79EPPZsllFIXFeYh7e0+sC2h+DT?= =?iso-8859-1?Q?CRYeJOb0taL3Zh5OXRCmwr5yGBEiPS8ZKex2FTwbo5HQPXvkqzrFQWEmhS?= =?iso-8859-1?Q?3R66vrqWu72YZa/Dl5gzPIFtpbe9tl9jdGVLF6z6irgBjPYuAkei7itAmU?= =?iso-8859-1?Q?+WWJ+8j90zqMG6nOd8qOIIoG990DT2s9he/VXqXdllsRhViAXFSK7qyUlm?= =?iso-8859-1?Q?OSV6R25neGDwxFbY7NccNSreAJyDBX6PRIkZwq2F9otZg5KHMT0vD+Kd+Y?= =?iso-8859-1?Q?l1Slp7CqvFIz7iXc4Wfb8XJWr9109BK4Yysp4cTxYFIipEn0mXJc1FjYy1?= =?iso-8859-1?Q?E7TATK5fc40PBDu9OkkuAeeoCGONpxXWkxivoxEg+X2+yfkvULHDiTVABC?= =?iso-8859-1?Q?qF5tSykW+yO3hb4Wm0tIww1/hVtSNp1tiZQGMNim1zC/VzVAkkQA3IoL0g?= =?iso-8859-1?Q?80m3SY1n/Ic4eSADY6Aqfh03zjm6kTgpjvFpovXk8YiG3dCc38diqbUj41?= =?iso-8859-1?Q?nv0O1DboHDb5GJt1QBK2Nrvt1VgF/sCfSF9tTpzcHm6kNhs6kwc7VJbeVx?= =?iso-8859-1?Q?0cfGSid5qllNgnAYh8n4Xj2E5c9wOr9ETlhjn87y+OHJ9wQaYbwTuP15+B?= =?iso-8859-1?Q?+7w9Olnv3uhrS2i4+jog8vI/JRNdA98lAYyRrLNMPo1AKJSpWsGH4YFKhc?= =?iso-8859-1?Q?qVig=3D=3D?= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: YQXPR0101MB0968.CANPRD01.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 9e2cfd5d-d191-44ef-ddb3-08d9b6b8f6d2 X-MS-Exchange-CrossTenant-originalarrivaltime: 03 Dec 2021 23:59:38.0258 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: fnLNwjOCAZJnKmiBChR47ReOWCZap6bdhUKc8ZtixLQ5TX5mxTc4NIn5uQAqMlVSDPfHEcQ0NkLbudb1HmyPSA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: QB1PR01MB2771 X-Rspamd-Queue-Id: 4J5VZp2j5Gz4ldd X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-ThisMailContainsUnwantedMimeParts: N Ryan Stone wrote:=0A= >=0A= > Set it to 4k and move on. I'm not aware of any efforts in the VM=0A= > layer or network stack to make >PAGE_SIZE clusters usable in practical=0A= > situations. Honestly, at this point I wish that we'd just kill them=0A= > entirely.=0A= I feel about the same. However, I'm guessing that there are some net=0A= drivers that can't work without them (at least for jumbo frames) and=0A= no one available to fix those drivers.=0A= =0A= At the least, we should strongly encourage net device driver authors=0A= to avoid their use wherever practicable.=0A= =0A= I also think that "don't use them" should be the default for device=0A= drivers that can optionally not use them.=0A= =0A= I'll admit I cannot think of a good way to fix jumbo clusters, so avoiding= =0A= them is about all we can do, I think?=0A= - Separate allocation pools would avoid the fragmentation problem,=0A= but waste a lot of space unless the mix of cluster sizes needed for the= =0A= network load on the machine was somehow known.=0A= =0A= rick=0A= =0A= On Fri, Dec 3, 2021 at 12:52 PM John Jasen wrote:=0A= >=0A= > Pretty close to two years ago, we tripped across conditions where heavily= =0A= > used FreeBSD 11.x packet-filter firewalls would slow to a crawl and load= =0A= > would go crazy.=0A= >=0A= > In a fit of hopefulness with our upgrade to FreeBSD 13.0, I removed the= =0A= > hw.cxgbe.largest_rx_cluster settings we put in place -- only to have our= =0A= > hardest hit firewall experience high load and plummeting bandwidth within= a=0A= > few hours.=0A= >=0A= > So, I have two questions -- is there a better fix for this? Or, is the=0A= > approach to set largest_rx_cluster to 4k and move on?=0A= >=0A= > Thanks in advance!=0A= =0A=