Lagg and multi-gigabit questions/proablems

From: Chris Ross <cross+freebsd_at_distal.com>
Date: Tue, 08 Nov 2022 05:38:54 UTC
Tl;dr; I have two FreeBSD systems attached to a Cisco switch, there should be multi-gigabit connectivity, but only seeing 1Gpbs.  Each system is trunked, vlan interfaces on the underlying interface.

I have an older Freebsd 11.x system with a 10gbe interface, There are a collection of vlan interfaces on it.
I have a newer Freebsd 12.3 system with lagg across two 1gbe interfaces.  There are a collection of vlan interfaces on the lagg.

I would _expect_ to be able to get 2gbps, or just shy thereof.  But, using iperf3 shows me the same ~940 Mbits/sec that I get when coming from a system a couple switches away on a single gigabit connection.

Looking on the Cisco switch, the port-channel (connected to the lagg) claims "BW 2000000 Kbit/sec”.  

Happy to share config with anyone who may be able to help, but won’t paste it all here.

                 - Chris