Diagnosing terrible ixl performance

[Garrett Wollman]

I'm commissioning a new NFS server with an Intel dual-40G XL710
interface, running 11.1.  I have a few other servers with this
adapter, although not running 40G, and they work fine so long as you
disable TSO.  This one ... not so much.  On the receive side, it gets
about 600 Mbit/s with lots of retransmits.  On the *sending* side,
though, it's not even able to sustain 10 Mbit/s -- but there's no
evidence of retransmissions, it's just sending really really slowly.
(Other machines with XL710 adapters are able to sustain full 10G.)
There is no evidence of any errors on either the adapter or the switch
it's connected to.

So far, I've tried:

- Using the latest Intel driver (no change)
- Using the latest Intel firmware (breaks the adapter)
- Disabling performance tweaks in loader.conf and sysctl.conf
- Changing congestion-control algorithms

Anyone have suggestions while I still have time to test this?  (My
plan B is to fall back to an X520 card that I have in my spares kit,
because I *know* those work great with no faffing about.)  Any
relevant MIBs to inspect?

The test I'm doing here is simple iperf over TCP, with MTU 9120.  It
takes about 10 seconds for the sending side to complete, but buffers
are severely constipated for 20 seconds after that (delaying all
traffic, including ssh connections).

I'm at the point of trying different switch ports just to eliminate
that as a possibility.

-GAWollman