strange performance receiving with ixgbe/82599

[Luigi Rizzo]

Jack,
i am doing some experiments with a modified ixgbe driver, and i am
seeing some strange performance numbers which I'd like to explain.

My card uses the 82599 chip, monted on an x16 slot on an Asus
motherboard. CPU is i7-870 @ 2.93GHz. Both -head and RELENG_8
exhibit the same behaviour. I am using a modified driver which
essentially accesses the rings from userspace, sending or
receiving frames as soon as there is room.

On the transmit side, using 64 byte packets (60+4 CRC) i can saturate
the link (measured 14.87-14.88 Mpps) with no problem with just one
core at 1.33 GHz and 2 or 4 queues (one queue tops at 12Mpps,
but that's not a big issue).

On the receive side, i can sustain 14.2Mpps (on 2-4 queues) if the
sender generates 68-byte packets (i.e. 64+4CRC), but as soon as i
go to 63+4 or less, the receive rate drops to 11.5-11.6Mpps.
CPU cycles abound, because the speed is unchanged down to 1.45GHz

Any reason to explain that suddend drop with short (but legal)
packet sizes ?

The stats counters report a lot of "missed" packets, though i
never see the receive rings become full. I was wondering if there
is any minimum inter-packet time that the receiver side needs,
but could not find any register that controls that.
The rings have 2048 slots each, so even something slow happening
when the rings wrap don't really explain such a huge (25% or so)
loss level.

cheers
luigi