ixl 40G bad performance?

[Luigi Rizzo]

On Monday, October 19, 2015, Eggert, Lars <lars at netapp.com> wrote:

> Hi,
>
> On 2015-10-19, at 16:20, Luigi Rizzo <rizzo at iet.unipi.it <javascript:;>>
> wrote:
> >
> > i would look at the following:
> > - c states and clock speed - make sure you never go below C1,
> >  and fix the clock speed to max.
> >  Sure these parameters also affect the 10G card, but there
> >  may be strange interaction that trigger the power saving
> >  modes in different ways
>
> I already have powerd_flags="-a max -b max -n max" in rc.conf, which I
> hope should be enough.


I suspect it might not touch the c states, but better check. The safest is
disable them in the bios.


>
> > - interrupt moderation (may affect ping latency,
> >  do not remember how it is set in ixl but probably a sysctl
>
> ixl(4) describes two sysctls that sound like they control AIM, and they
> default to off:
>
> hw.ixl.dynamic_tx_itr: 0
> hw.ixl.dynamic_rx_itr: 0
>
>
There must be some other control for the actual (fixed, not dynamic)
moderation.


> > - number of queues (32 is a lot i wouldn't use more than 4-8),
> >  may affect cpu-socket affinity
>
> With hw.ixl.max_queues=4 in loader.conf, performance is still unchanged.
>
> > - tso and flow director - i have seen bad effects of
> >  accelerations so i would run the iperf test with
> >  of these features disabled on both sides, and then enable
> >  them one at a time
>
> No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro".
>
> How do I turn off flow director?


I am not sure if it is enabled I'm FreeBSD. It is in linux and almost
halves the pkt rate with netmap (from 35 down to 19mpps).
Maybe it is not too bad for bulk TCP.


>
> > - queue sizes - the driver seems to use 1024 slots which is
> >  about 1.5 MB queued, which in turn means you have 300us
> >  (and possibly half of that) to drain the queue at 40Gbit/s.
> >  150-300us may seem an eternity, but if a couple of cores fall
> >  into c7 your budget is gone and the loss will trigger a
> >  retransmission and window halving etc.
>
> Also no change with "hw.ixl.ringsz=256" in loader.conf.


Any better success with 2048 slots?
3.5 gbit  is what I used to see on the ixgbe with tso disabled, probably
hitting a CPU bound.

Cheers
Luigi


> This is really weird.
>
> Lars
>


-- 
-----------------------------------------+-------------------------------
 Prof. Luigi RIZZO, rizzo at iet.unipi.it  . Dip. di Ing. dell'Informazione
 http://www.iet.unipi.it/~luigi/        . Universita` di Pisa
 TEL      +39-050-2217533               . via Diotisalvi 2
 Mobile   +39-338-6809875               . 56122 PISA (Italy)
-----------------------------------------+-------------------------------