Lagg-LACP is not working with Intel Fortville NIC on FreeBSD HOL (11.0 Current)

[Pushkar Kothavade]

Dear Members,

I am part of Panasas. I am evaluating 'Lagg-LACP performance and 
robustness' over Intel Fortville NIC (IXL) on Intel Taylor Pass platform 
with FreeBSD-HOL [11-CURRENT] installed. Lagg-LACP feature works fine 
(with satisfactory performance) when Promiscuous mode is enabled on the 
Lagg.
If Promiscuous mode is disabled, in that case even basic Lagg 
functionality does not work.

*Setup Information : *

- Machine (Intel Taylor Pass) is running FreeBSD-HOL (11 current)
- Machine has 'Intel Fortville NIC (Intel Ethernet Controller XL710)' 
installed with special firmware to support four 10Gbps ports with 
breakout cable
- Breakout cable is used with one of the 40GB ports which creates four 
10GB ports
- All four 10Gbps ports are connected to Dell Force 10 Switch
- Lagg-LACP has been configured on all four 10GB ports of the Machine 
and on Switch ports as well

*Problem Statement :*

# When machine sends an Arp-request then interface does not receive 
Arp-reply
    - Arp request sent by the machine is successful
    - Switch is able to get Arp broadcast request sent by the machine
    - Switch sends back unicast Arp-reply to machine
    - But still driver does not receive Arp-reply

# Even after adding static Arp entry in machine's Arp table, ping fails

# Configured same MAC address on all 4 interfaces which are part of 
Lagg, still ping fails
     Is there any problem with MAC filtering ?

*Workaround : *

- After putting Lagg interface in promiscuous mode, network interface 
seems to work properly.

*Platform Information :*

Machine - Intel Taylor Pass server board
OS - FreeBSD HOL (11.0 Current)
NIC - Intel Fortville NIC (Intel Ethernet Controller XL710) - Find more 
information as below
Switch - Dell Force 10 Switch

<Snip>
ixl0: <Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.1> mem 
0x383ffe800000-0x383ffeffffff,0x383fff818000-0x383fff81ffff irq 42 at 
device 0.0 on pci4
ixl0: Using MSIX interrupts with 13 vectors
ixl0: f4.33 a1.2 n04.42 e80001935
ixl0: Ethernet address: 68:05:ca:35:97:28
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl1: <Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.1> mem 
0x383ffe000000-0x383ffe7fffff,0x383fff810000-0x383fff817fff irq 42 at 
device 0.1 on pci4
ixl1: Using MSIX interrupts with 13 vectors
ixl1: f4.33 a1.2 n04.42 e80001935
ixl1: Ethernet address: 68:05:ca:35:97:29
ixl1: PCI Express Bus: Speed 8.0GT/s Width x8
ixl2: <Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.1> mem 
0x383ffd800000-0x383ffdffffff,0x383fff808000-0x383fff80ffff irq 42 at 
device 0.2 on pci4
ixl2: Using MSIX interrupts with 13 vectors
ixl2: f4.33 a1.2 n04.42 e80001935
ixl2: Ethernet address: 68:05:ca:35:97:2a
ixl2: PCI Express Bus: Speed 8.0GT/s Width x8
ixl3: <Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.1> mem 
0x383ffd000000-0x383ffd7fffff,0x383fff800000-0x383fff807fff irq 42 at 
device 0.3 on pci4
ixl3: Using MSIX interrupts with 13 vectors
ixl3: f4.33 a1.2 n04.42 e80001935
ixl3: Ethernet address: 68:05:ca:35:97:2b
ixl3: PCI Express Bus: Speed 8.0GT/s Width x8
acpi0: Power Button (fixed)
ixl0: <Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.1> mem 
0x383ffe800000-0x383ffeffffff,0x383fff818000-0x383fff81ffff irq 42 at 
device 0.0 on pci4
ixl0: Using MSIX interrupts with 13 vectors
ixl0: f4.33 a1.2 n04.42 e80001935
ixl0: Ethernet address: 68:05:ca:35:97:28
ixl0: PCI Express Bus: Speed 8.0GT/s Width x8
ixl1: <Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.1> mem 
0x383ffe000000-0x383ffe7fffff,0x383fff810000-0x383fff817fff irq 42 at 
device 0.1 on pci4
ixl1: Using MSIX interrupts with 13 vectors
ixl1: f4.33 a1.2 n04.42 e80001935
ixl1: Ethernet address: 68:05:ca:35:97:29
ixl1: PCI Express Bus: Speed 8.0GT/s Width x8
ixl2: <Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.1> mem 
0x383ffd800000-0x383ffdffffff,0x383fff808000-0x383fff80ffff irq 42 at 
device 0.2 on pci4
ixl2: Using MSIX interrupts with 13 vectors
ixl2: f4.33 a1.2 n04.42 e80001935
ixl2: Ethernet address: 68:05:ca:35:97:2a
ixl2: PCI Express Bus: Speed 8.0GT/s Width x8
ixl3: <Intel(R) Ethernet Connection XL710 Driver, Version - 1.4.1> mem 
0x383ffd000000-0x383ffd7fffff,0x383fff800000-0x383fff807fff irq 42 at 
device 0.3 on pci4
ixl3: Using MSIX interrupts with 13 vectors
ixl3: f4.33 a1.2 n04.42 e80001935
ixl3: Ethernet address: 68:05:ca:35:97:2b
ixl3: PCI Express Bus: Speed 8.0GT/s Width x8
<Snip>

Thanks,
Pushkar Kothavade