Chelsio T520-SO-CR low performance (netmap tested) for RX

[Navdeep Parhar]

On Sat, Jan 23, 2016 at 11:12:28AM -0800, Luigi Rizzo wrote:
> On Sat, Jan 23, 2016 at 10:38 AM, Navdeep Parhar <nparhar at gmail.com> wrote:
> > On Sat, Jan 23, 2016 at 03:48:39PM -0200, Marcus Cenzatti wrote:
> > ...
> >>
> >> woops, my bad, yes probably we had some drop, with -S and -D now I get 1.2Mpps.
> >
> > Run "netstat -hdw1 -i cxl<n>" on the receiver during your test.
> 
> Navdeep, does this give any info on the ncxl port rather
> than the cxl port connected to the host stack ?

You're right, it should have been "netstat -hdw 1 -I ncxl<n>".  In these
kinds of experiments it might even be best to run two netstats in
parallel on cxl and ncxl.

> 
> ...
> > Do you know if the transmitter will pad up so as not to put runts on the
> > wire?  If not then you might want to bump up the size of the frame
> > explicitly (there's some pkt-gen knob for this).
> >
> 
> ix/ixl do automatic padding, and in any case pkt-gen
> by default generates valid packet sizes (and so it does
> with the variable-size tests I suggested).
> 
> Is there any parameter that controls interrupt moderation ?
> 
> In any case we need to know the numbers when sending to the
> ncxl MAC address as opposed to broadcast.
> 
> I suspects one of these problems:
> 
> - the interrupt moderation interval is too long thus limiting
>   the rate to one ring/interval. Unlikely though, even
>   with 1k slots, the measured 1.2 Mpps corresponds to almost
>   1ms which is too long
> 
> - the receiver cannot cope with the input load and somehow
>   takes a long time to recover from congestion. If this is
>   the case, running the sender at a lower rate might reach
>   a peak throughput > 1.2 Mpps when the receiver can still
>   keep up, and then drop to the low rate under congestion.
> 
> - and of course bus errors, when the device is connected on
>   a PCIe slot with only 1-2 data lanes.
>   This actually happens a lot, physical connector sizes
>   do not reflect the number of active PCIe lanes.

There are no drops or PAUSE or any sign of backpressure.  The netstat
counters show 900K incoming and 0 drops/errors, which means 900K packets
on the wire for the port and all were delivered to the driver
successfully.

The mismatch in the transmitter's counter and the incoming counter can
only be explained by
a) Frames whose DMAC address didn't match the local interface's MAC.
This can be tested by switching cxl0 and ncxl0 to promisc mode to see if
that opens the flood gates.
b) Frames mangled badly enough to be discarded.  But these should show
as an error or drop in at least one of these:

sysctl dev.cxl.<n>.stats
sysctl -n dev.t5nex.0.misc.tp_err_stats
netstat -hd -I cxl0
netstat -hd -I ncxl0

The only broken counter in cxgbe that I know of is rx_runts and we've
already verified that the transmitter isn't generating runts.

Regards,
Navdeep