Diagnosing terrible ixl performance

[Rick Macklem]

I don't know if this post is helpful, but just in case:
http://docs.FreeBSD.org/cgi/mid.cgi?04125f40-6388-f074-d935-ce6c16d220fa

Hope you don't mind a top post, rick

________________________________________
From: owner-freebsd-net at freebsd.org <owner-freebsd-net at freebsd.org> on behalf of hiren panchasara <hiren at strugglingcoder.info>
Sent: Friday, April 20, 2018 5:20:45 PM
To: Garrett Wollman
Cc: freebsd-net at freebsd.org
Subject: Re: Diagnosing terrible ixl performance

On 04/20/18 at 12:03P, Garrett Wollman wrote:
> I'm commissioning a new NFS server with an Intel dual-40G XL710
> interface, running 11.1.  I have a few other servers with this
> adapter, although not running 40G, and they work fine so long as you
> disable TSO.  This one ... not so much.  On the receive side, it gets
> about 600 Mbit/s with lots of retransmits.  On the *sending* side,
> though, it's not even able to sustain 10 Mbit/s -- but there's no
> evidence of retransmissions, it's just sending really really slowly.
> (Other machines with XL710 adapters are able to sustain full 10G.)
> There is no evidence of any errors on either the adapter or the switch
> it's connected to.
>
> So far, I've tried:
>
> - Using the latest Intel driver (no change)
> - Using the latest Intel firmware (breaks the adapter)
> - Disabling performance tweaks in loader.conf and sysctl.conf
> - Changing congestion-control algorithms
>
> Anyone have suggestions while I still have time to test this?  (My
> plan B is to fall back to an X520 card that I have in my spares kit,
> because I *know* those work great with no faffing about.)  Any
> relevant MIBs to inspect?
>
> The test I'm doing here is simple iperf over TCP, with MTU 9120.  It
> takes about 10 seconds for the sending side to complete, but buffers
> are severely constipated for 20 seconds after that (delaying all
> traffic, including ssh connections).
>
> I'm at the point of trying different switch ports just to eliminate
> that as a possibility.

You are already trying to check if the switch in-between isn't causing
the problem. A few other (probably obvious) things to try:
- sysctl -a | grep hw.ixl or dev.ixl to see if you find anything useful
(actual name might not be ixl, but you get the point)
- Try with lower mtu to see if that's causing anything interesting
- If you can reproduce easily, a single stream pcap might be useful from
  both send and recv side to understand the slowness.

Cheers,
Hiren