solved: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX

Luigi Rizzo rizzo at iet.unipi.it
Sun Jan 24 07:23:05 UTC 2016


On Sat, Jan 23, 2016 at 10:42 PM, Navdeep Parhar <nparhar at gmail.com> wrote:
> On Sat, Jan 23, 2016 at 09:33:32PM -0800, Luigi Rizzo wrote:
>> On Sat, Jan 23, 2016 at 8:28 PM, Marcus Cenzatti <cenzatti at hush.com> wrote:
>> >
>> >
>> > On 1/24/2016 at 1:10 AM, "Luigi Rizzo" <rizzo at iet.unipi.it> wrote:
...
>> One last attempt: try use -l 64 on the sender, this will generate 64+4 byte
>> packets, which may become just 64 on the receiver if the chelsio is configured
>> to strip the CRC. This should result in well aligned PCIe transactions and
>> reduced PCIe traffic, which may help (the ix driver has a similar problem,
>> but since it does not strip the CRC can rx at line rate with 60 bytes but not
>> with 64).
>
> Keep hw.cxgbe.fl_pktshift in mind for these kind of tests.  The default
> value is 2 so the chip DMAs payload at an offset of 2B from the start of
> the rx buffer.  So you'll need to adjust your frame size by 2 (66B on
> the wire, 62B after CRC is removed, making it exactly 64B across PCIe if
> pktshift is 2) or just set hw.cxgbe.fl_pktshift=0 in /boot/loader.conf.

This sounds like something to fix.

In netmap packets are supposed to be aligned with the
beginning of the buffer, so when operating in netmap
mode at least the pktshift should be set to 0.
If it is not possible to have it per- queue, I would
suggest to set it to 0 unconditionally when you compile
a netmap-enabled kernel.

How much of a performance boost do you see with a
shift of 2 with regular traffic ? Modern CPUs seem
to be pretty good with unaligned memory accesses.

cheers
luigi

> Regards,
> Navdeep



-- 
-----------------------------------------+-------------------------------
 Prof. Luigi RIZZO, rizzo at iet.unipi.it  . Dip. di Ing. dell'Informazione
 http://www.iet.unipi.it/~luigi/        . Universita` di Pisa
 TEL      +39-050-2217533               . via Diotisalvi 2
 Mobile   +39-338-6809875               . 56122 PISA (Italy)
-----------------------------------------+-------------------------------


More information about the freebsd-net mailing list