ix(intel) vs mlxen(mellanox) 10Gb performance

[Adrian Chadd]

Hi,

Some hand-waving suggestions:

* if you're running something before 10.2, please disable IXGBE_FDIR
in sys/conf/options and sys/modules/ixgbe/Makefile . It's buggy and it
caused a lot of issues.
* It sounds like some extra latency is happening, so I'd fiddle around
with interrupt settings. By default it does something called adaptive
interrupt moderation and it may be getting in the way of what you're
trying to do. There's a way to disable AIM in /boot/loader.conf and
manually set the interrupt rate.
* As others have said, TSO has been a bit of a problem - hps has been
working on solidifying the TSO configuration side of things so NICs
advertise to the stack what their maximum offload capability is so
things like NFS and TCP don't exceed the segment count. I don't know
if it's tunable without hacking the driver, but maybe hack the driver
to reduce the count a little to make sure you're not overflowing
things and causing it to fall back to a slower path (where it copies
all the mbufs into a single larger one to send to the NIC.)
* Disable software LRO and see if it helps. Since you're doing lots of
little non-streaming operations, it may actually be hindering.

HTH,


-adrian