Intel NETMAP performance and packet type

[Slawa Olhovchenkov]

On Fri, Feb 28, 2020 at 06:37:22PM +0100, Vincenzo Maffione wrote:

> Il giorno ven 28 feb 2020 alle ore 12:26 Slawa Olhovchenkov <slw at zxy.spb.ru>
> ha scritto:
> 
> > On Thu, Feb 27, 2020 at 11:16:50PM +0300, Slawa Olhovchenkov wrote:
> >
> > > On Thu, Feb 27, 2020 at 06:51:54PM +0100, Vincenzo Maffione wrote:
> > >
> > > > Hi,
> > > >   So, the issue is not the payload.
> > > > If you look at the avg_batch statistics reported by pkt-gen, you'll see
> > > > that in the ACK-flood experiment you have 4.92, whereas in the
> > SYN-flood
> > > > case you have 17.5. The batch is the number of packets (well, actually
> > > > netmap descriptors, but in this case it's the same) that you receive
> > (or
> > > > transmit) for each poll() invocation.
> > > > So in the first case you end up doing much more poll() calls, hence the
> > > > higher per-packet overhead and the lower packet-rate.
> > > >
> > > > Why is the poll() called more frequently? That depends on packet
> > timing and
> > > > interrupt rate. There must be something different on your packet
> > generator
> > > > that produces this effect (e.g. different burstiness, or maybe the
> > packet
> > > > generator is not able to saturate the 10G link)?
> > >
> > > No, I am capture netstat output -- raw packet rate is the same.
> > > Also, I am change card to chelsio T5 and don't see issuse.
> > >
> > > This is payload issuse, at driver level.
> > >
> > > > In any case, I would suggest measuring the RX interrupt rate, and check
> > > > that it's higher in the ACK-flood case. Then you can try to lower the
> > > > interrupt rate by tuning the interrupt moderation features of the
> > Intel NIC
> > > > (e,g. limit hw.ix.max_interrupt_rate and disable hw.ix.enable_aim or
> > > > similar).
> > > > By playing with the interrupt moderation you should be able to
> > increase the
> > > > avg_batch, and then increase throghput.
> > >
> > > Already limited.
> >
> > Also, is this normal (rxd_tail == rxd_head):
> >
> > dev.ix.0.queue0.rx_discarded: 0
> > dev.ix.0.queue0.rx_copies: 0
> > dev.ix.0.queue0.rx_bytes: 612041623304
> > dev.ix.0.queue0.rx_packets: 9563149414
> > dev.ix.0.queue0.rxd_tail: 1120
> > dev.ix.0.queue0.rxd_head: 1120
> > dev.ix.0.queue0.irqs: 40154885
> > dev.ix.0.queue0.interrupt_rate: 16129
> > dev.ix.0.queue0.tx_packets: 553897984
> > dev.ix.0.queue0.tso_tx: 0
> > dev.ix.0.queue0.txd_tail: 0
> > dev.ix.0.queue0.txd_head: 0
> >
> > I am see this RX queue is stoped.
> >

Ah, may fault. This is different case on same Intel card, this is no
ACK-flood.

And I am see this on production traffic.

> Yes, (rxd_tail == rxd_head) means that the NIC ran out of RX buffers.
> rxd_head is the next descriptor that the NIC will use. rxd_tail is the next
> descriptor that the driver will replenish. RX buffers are replenished by
> the netmap NIOCRXSYNC routine, which is called on poll().

poll() still called frequenced but rxd_head/rxd_tail stalled.

> However, rx_discarded is 0, which means that the NIC is not dropping
> packets. So the problem should not be that poll() is not called frequently
> enough.

poll() called for all queue synchronously for multiple queue, stalled
only one.

> You should check rx_discarded for all the queues.

All zero.

> Another thing you need to check is how the load is balanced across the
> receive queues. How many have you configured? Maybe the two workloads
> (SYN-flood and ACK-flood) load different queues in different ways.

Sorry, this is different case: after some time (after hour, for
example) some queue stalled infinitly. Rest queue handle traffic.

This is Intel card, iflib variant.