From nobody Tue Nov 08 08:59:00 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4N628m1sCnz4h3yn for ; Tue, 8 Nov 2022 08:59:20 +0000 (UTC) (envelope-from lutz@iks-jena.de) Received: from annwfn.iks-jena.de (annwfn.iks-jena.de [IPv6:2001:4bd8::19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4N628l1Z49z3sW2 for ; Tue, 8 Nov 2022 08:59:19 +0000 (UTC) (envelope-from lutz@iks-jena.de) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of lutz@iks-jena.de designates 2001:4bd8::19 as permitted sender) smtp.mailfrom=lutz@iks-jena.de; dmarc=none X-SMTP-Sender: IPv6:2001:4bd8:0:666:248:54ff:fe12:ee3f Received: from belenus.iks-jena.de (belenus.iks-jena.de [IPv6:2001:4bd8:0:666:248:54ff:fe12:ee3f]) by annwfn.iks-jena.de (8.15.2/8.15.2) with ESMTPS id 2A88x0NS001218 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 8 Nov 2022 09:59:01 +0100 X-MSA-Host: belenus.iks-jena.de Received: (from lutz@localhost) by belenus.iks-jena.de (8.14.3/8.14.1/Submit) id 2A88x0NQ021251; Tue, 8 Nov 2022 09:59:00 +0100 Date: Tue, 8 Nov 2022 09:59:00 +0100 From: Lutz Donnerhacke To: Chris Ross Cc: freebsd-net@freebsd.org Subject: Re: Lagg and multi-gigabit questions/proablems Message-ID: <20221108085900.GA21023@belenus.iks-jena.de> References: <76E0E023-AB86-453A-B95C-AED44E7ADA2D@distal.com> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <76E0E023-AB86-453A-B95C-AED44E7ADA2D@distal.com> X-message-flag: Please send plain text messages only. Thank you. User-Agent: Mutt/1.5.17 (2007-11-01) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.10 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.997]; FORGED_SENDER(0.30)[lutz@donnerhacke.de,lutz@iks-jena.de]; R_SPF_ALLOW(-0.20)[+ip6:2001:4bd8::/48]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[2001:4bd8:0:666:248:54ff:fe12:ee3f:received]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; ASN(0.00)[asn:15725, ipnet:2001:4bd8::/29, country:DE]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[donnerhacke.de]; TAGGED_RCPT(0.00)[freebsd]; FROM_NEQ_ENVFROM(0.00)[lutz@donnerhacke.de,lutz@iks-jena.de] X-Rspamd-Queue-Id: 4N628l1Z49z3sW2 X-ThisMailContainsUnwantedMimeParts: N On Tue, Nov 08, 2022 at 12:38:54AM -0500, Chris Ross wrote: > Tl;dr; I have two FreeBSD systems attached to a Cisco switch, there should be multi-gigabit connectivity, but only seeing 1Gpbs. Each system is trunked, vlan interfaces on the underlying interface. I can add another data point: - stable trains in 12/13 - 10g ixl cards (with 1G and 10G ports) - without LACP cpu load correlates linearly with the throughput - with LACP cpu load rise sharply to 100% at about 40-50% input bandwidth - if this happens, the packet drop rate spikes to about 10 kp/s in the kernel input handling queues, throughput falls to some Mbps Unfortunately I did not have time to digg deeper, bringing the production network back to work had priority. (The BSD machines in question work as cantral Carrier Grade NAT devices.)