Network throughput not reaching line rate. Need clarification on iflib.

[Rajesh Kumar]

Hi FreeBSD team,

I am writing a network driver using iflib framework and using "iperf3" tool
for performance testing.

1) 10G link with 9000 MTU. Seeing line-rate numbers on both Tx and Rx path.
2) 10G link with 1500 MTU. Seeing line-rate numbers on Tx path, but low
numbers on Rx path.

I suspect the driver design here, but need some clarification on the iflib
framework.

My driver expects two memory buffers per receive descriptor. So I used "2
fls" per rxq (means isc_nrxq = 1, and isc_nfl = 2). For every descriptor, I
use the memory buffers at same index from both the free-lists to refill my
descriptors.  Also, I return the fragments with appropriate index and flid
from rxd_pkt_get. This approach works for me. But I am facing performance
issue as stated above. So, Is this approach perfect? or any better way of
doing?

Also, I did a quick analysis of tcpdump on both iperf3 server (my test
side) and client side(support side).  iperf3 clients pumps traffic to
iperf3 server. After a point, server sends an ACK with SACK option (SLE and
SRE) to indicate missing frame to the client. And client is re-transmitting
that segment as per protocol and the server receives it. But still I see
the ACK sent again with SACK option. So client keeps on re-transmitting the
packets and hence the performance drops.

Any reason why server is not acknowledging the re-transmitted packet? If
the driver Rx isn't proper, then I may not see the packet in tcpdump. But I
see it. So any idea, what could be wrong here?

Please let me know if any more details are needed.

Thanks,
Rajesh.