[Bug 248652] iflib: netmap pkt-gen large TX performance difference between 11-STABLE and 12-STABLE/CURRENT on ix & ixl NIC

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248652

--- Comment #31 from Vincenzo Maffione <vmaffione at FreeBSD.org> ---
(In reply to Krzysztof Galazka from comment #30)
Hi Krzysztof,
  I agree, and created a separate review for this possible change:
https://reviews.freebsd.org/D26896
It would be nice if you guys could run some tests to validate the change
against normal TCP/IP stack usage (e.g. non-netmap).

Speaking about the non-iflib driver, I guess it is acceptable for the ixl_xmit
routine to always set the report flag on the EOP packet.
However, this is not acceptable for netmap, and indeed the non-iflib netmap ixl
driver is only setting it twice per ring (see
https://github.com/freebsd/freebsd/blob/stable/11/sys/dev/netmap/if_ixl_netmap.h#L221-L223).
This is, in my opinion, what explains the huge difference in performance
between non-iflib and iflib for ixl.

If my understanding is correct, and according to our past experience with
netmap, the report flag will cause the NIC to initiate a DMA transaction to
either set the DD bit in the descriptor, or perform a memory write to update
the shadow TDH.
This is particularly expensive in netmap when done for each descriptor,
specially because netmap uses single-descriptor packets.
Interrupt moderation can also help a lot to mitigate the CPU overhead, but as
far as I see it does not limit the writeback DMA transactions, and therefore it
does not help in the netmap use-case. Moreover, my understanding is that iflib
is not using hardware interrupts on ixl (nor ix), but rather is using
"softirqs", so I guess that interrupt moderation does not play a role here. I
may be wrong on this last point.

I suspect the 1-queue pkt-gen hang problem may be due to something else, rather
than the report flag change. As a matter of facts, the same logic was working
fine on the non-iflib driver.

-- 
You are receiving this mail because:
You are on the CC list for the bug.