ixl 40G bad performance?

[Eric Joyner]

Bruce mostly has it right -- ITR is the minimum latency between interrupts.
But, it does actually guarantee a minimum period between interrupts.
Though, Fortville actually is unique a little bit in that there is another
ITR setting that can ensure a certain average number of interrupts per
second (called Interrupt Rate Limiting), though, but I don't think this is
used in the current version of the driver.

I see that the sysctl does clobber the global value, but have you tried
lowering the interval / raising the rate? You could try something like
10usecs, and see if that helps. We'll do some more investigation here --
3Gb/s on a 40Gb/s using default settings is terrible, and we shouldn't let
that be happening.

- Eric

On Thu, Oct 22, 2015 at 10:36 PM Bruce Evans <brde at optusnet.com.au> wrote:

> On Wed, 21 Oct 2015, Bruce Evans wrote:
>
> > Fix for em:
> >
> > X diff -u2 if_em.c~ if_em.c
> > X --- if_em.c~        2015-09-28 06:29:35.000000000 +0000
> > X +++ if_em.c 2015-10-18 18:49:36.876699000 +0000
> > X @@ -609,8 +609,8 @@
> > X         em_tx_abs_int_delay_dflt);
> > X     em_add_int_delay_sysctl(adapter, "itr",
> > X -       "interrupt delay limit in usecs/4",
> > X +       "interrupt delay limit in usecs",
> > X         &adapter->tx_itr,
> > X         E1000_REGISTER(hw, E1000_ITR),
> > X -       DEFAULT_ITR);
> > X +       1000000 / MAX_INTS_PER_SEC);
> > X X   /* Sysctl for limiting the amount of work done in the taskqueue */
> >
> > "delay limit" is fairly good wording.  Other parameters tend to give long
> > delays, but itr limits the longest delay due to interrupt moderation to
> > whatever the itr respresents.
>
> Everything in the last paragraph is backwards (inverted).  Other
> parameters tend to give short delays.  They should be set to small
> values to minimise latency.  Then under load, itr limits the interrupt
> _rate_ from above.  The interrupt delay is the inverse of the interrupt
> rate, so it is limited from below.  So "delay limit" is fairly bad
> wording.  Normally, limits are from above, but the inversion makes
> the itr limit from below.
>
> This is most easily understood by converting itr to a rate: itr = 125
> means a rate limit of 8000 Hz.  It doesn't quite mean that the latency
> is at least 125 usec.  No one wants to ensure large latencies, and the
> itr setting only ensures a minimal average latency them under load.
>
> Bruce
>