strange performance receiving with ixgbe/82599

[Ben Hutchings]

On Mon, 2011-05-23 at 19:20 +0200, Luigi Rizzo wrote:
> Jack,
> i am doing some experiments with a modified ixgbe driver, and i am
> seeing some strange performance numbers which I'd like to explain.
> 
> My card uses the 82599 chip, monted on an x16 slot on an Asus
> motherboard. CPU is i7-870 @ 2.93GHz. Both -head and RELENG_8
> exhibit the same behaviour. I am using a modified driver which
> essentially accesses the rings from userspace, sending or
> receiving frames as soon as there is room.
> 
> On the transmit side, using 64 byte packets (60+4 CRC) i can saturate
> the link (measured 14.87-14.88 Mpps) with no problem with just one
> core at 1.33 GHz and 2 or 4 queues (one queue tops at 12Mpps,
> but that's not a big issue).
> 
> On the receive side, i can sustain 14.2Mpps (on 2-4 queues) if the
> sender generates 68-byte packets (i.e. 64+4CRC), but as soon as i
> go to 63+4 or less, the receive rate drops to 11.5-11.6Mpps.
> CPU cycles abound, because the speed is unchanged down to 1.45GHz
> 
> Any reason to explain that suddend drop with short (but legal)
> packet sizes ?
[...]

With 64-bit wide RAM, a DMA write of 64 bytes with 8-byte alignment
covers exactly 8 memory words.  However a DMA write of 63 bytes with
8-byte alignment at the start covers 7 full and one partial memory word.
So a read-modify-write operation is required for the last word.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.