cxgbe netmap promiscuous mode?

[Joe Jones]

Hello,

We have a T520-SO and have made a new install of 11.0, to begin with the 
box would panic every time we tried to switch the card into netmap mode. 
So we recompiled the kernel with netmap removed, then compiled the 
netmap kernel module from github, as this in our experience generally 
leads to a more stable netmap.

we have

uname -a
FreeBSD goose2 11.0-RELEASE-p1 FreeBSD 11.0-RELEASE-p1 #0: Wed Mar 22 
16:52:35 UTC 2017     joe at goose2:/usr/obj/usr/src/sys/GENERIC amd64

and the following in /boot/loader.conf

t4fw_cfg_load="YES"
t5fw_cfg_load="YES"
if_cxgbe_load="YES"
hw.cxgbe.fl_pktshift=0
hw.cxgbe.toecaps_allowed=0
hw.cxgbe.nnmtxq10g=8
hw.cxgbe.nnmrxq10g=8
hw.cxgbe.num_vis=2

Before I run our application I run

ifconfig cxl1 promisc -vlanhwtag up

Now our application can now start without panicking the kernel. Here is 
where it gets interesting, our application is able to announce it's self 
via ARP, I can see the ethernet switch learning which port it's on, and 
other hosts adding it to their ARP tables. When I try an ICMP ping it 
goes missing. After watching the TX packet graph for the connected port 
on the switch while starting and stopping a flood ping to the 
application, I'm sure the packets are getting sent to the card, however 
I don't see them in the netmap ring. If I kill our application, then use 
ifconfig to create and configure a vlan port I can confirm that the card 
is working and has connectivity.

Here's what I think is happening. ARP requests are received because they 
are sent to the broadcast address. Our application then announces it's 
self. However traffic destined for the application is send to a MAC 
address which is neither the broadcast or the MAC programed into the 
hardware and is dropped. My understanding of promiscuous it that it 
informs the card that we want these dropped packets. It looks to me 
like, when the card is in netmap mode the promisc flag is being ignored.

I have also tried using freebsd-update to update to p8. As with the p0 
kernel we get a panic when we switch the card into netmap mode.

We did previously have these cards working in netmap mode. We were using 
a pre 11 snapshot of the svn head though .

Many Thanks

Joe Jones
Stream Technologies