Re: Avoiding the lagg(4) bottleneck

On Sat, Apr 26, 2025 at 2:34 PM Brett Glass <brett@lariat.net> wrote:
>
> All:
>
> The LAG configuration is about as simple as it can be:
>
> ifconfig_igb2="up -tso4 -tso6 -lro -vlanhwtso media 1000baseTX
> mediaopt full-duplex"
> ifconfig_igb3="up -tso4 -tso6 -lro -vlanhwtso media 1000baseTX
> mediaopt full-duplex"
> cloned_interfaces="lagg0"
> ifconfig_lagg0="laggproto lacp laggport igb2 laggport igb3 lacp_fast_timeout"
>
> Other sysctls of possible interest (from /boot/loader.conf):
>
> # Disable hyperthreading
> machdep.hyperthreading_allowed="0"
> # Allowing multithreaded networking yields major load reduction
> net.isr.maxthreads="-1"
> # Optimizations for NICs
> # This is claimed to provide improvement with em/igb driver
> hw.em.rx_process_limit="-1"
> # Don't be overwhelmed by interrupts
> hw.em.max_interrupt_rate=10000
> # Don't try to save energy on em/igb interface
> hw.em.eee_setting=0
>
> We have tried
>
> net.link.lagg.default_use_flowid=0
>
> but it does not seem to matter.
>
> Our concern is that the lagg(4) driver may be limited by single
> threading and/or locks.
> We are testing with real life loads, attempting to saturate a pair of gigabit
> upstream connections. Try as we might, we can't get 2 Gbps out of
> it; in fact, we
> have trouble getting 1 Gbps out of the pair.
>
> --Brett Glass
>
> At 10:05 PM 4/20/2025, Paul Procacci wrote:
>
> >What is your lagg configuration?
> >How many different ip/port source tuples are involved with your testing?
> >How are you testing?
> >
> >There's no known deficiency with FreeBSD's lagg driver.
> >
> >~Paul
> >
> >--
> >__________________
> >
> >:(){ :|:& };:
>

How many source ip/port tuples are involved in your testing?
With just a single source, you won't ever exceed a 1G link regardless
of how many links there are.

Where is the source data coming from?  If coming from rust, you won't
saturate a 1G link.

I have multiple lagg setups where I work (finance) and can easily
saturate aggregated 1G nics.
Now each of my links land on separate switches with mlagg doing its
thing, but that itself matters not and only entices the following
question.  Are your switches dropping anything?  Are your switches
pausing received frames? ... etc?

You need to run a netperf from multiple sources with this lagg machine
as a sink.  This would answer a lot of the above questions.
While running netperf you have to run top with -SHP (I think are the
flags) to capture system related information.  Shuffling 1G around is
a snore for FreeBSD.

~Paul
-- 
__________________

:(){ :|:& };: