swaping ring slots between NIC ring and Host ring does not always success

[Xiaoye Sun]

Hi all,

I wrote a netmap application that is very similar to bridge.c in master/example
directory of the netmap code. The major difference between my program and
bridge.c is that my application also creates custom packets that are
directly put into the tx ring of the NIC (like a packet generator with
NIC-Host packet forwarding). Each generated packet has an unique sequence
number so that the receiver can tell if a packet is lost.

Like bridge.c, the packet forwarding between the nic and the host uses
'zerocopy', so that the program only swaps the buf_idx of the slots to
virtually forward the packet to the other end. NS_BUF_CHANGED is set for
the slots (both rx and tx) whose buf_idx has been changed due to zerocopy.

I set the number of rings on nic to 1 using the ethtool command, i.e., one
pair of netmap rings for the nic (one for host tx and one for host rx) and
another pair of rings for the host (one for host tx and one for host rx).

However, at the receiver side, I found that there is a chance (very little
chance, less than 1% of the swaps) where swapping the buf_idx does not
success. The receiver might get the packet in the buffer swapped out. For
example, the netmap program wants to swap host slot *SH* with NIC slot *SN*
in order to send the packet in *SH* from host to the receiver; the receiver
might get the packet in *SN* instead. This usually happens to the very end
of the available slots.

Our experiment is done on *Linux* machine (Linux 3.16.0-4-amd64 #1 SMP
Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux) with* intel NIC*
(Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection) using
the *ixgbe* driver.

To understand this problem, I look into the patch code to the ixgbe driver.
I found that the flag NS_BUF_CHANGED is not used in the linux driver for
ixgbe. please look the header file /LINUX/ixgbe_netmap_linux.h. In funtions
ixgbe_netmap_txsync and ixgbe_netmap_rxsync, the function call
netmap_reload_map is commented out. However, the ixgbe's driver patch for
FreeBSD calls the reload function.

So, I am wondering if this is a known problem when using netmap on LINUX.
Has anybody found the same problem?
Is netmap_reload_map required in Linux? why it is not called in the driver
patch?
What is the recommended solution for this problem?

Thanks!

Best,
Xiaoye