Bridge Not Forwarding ARP

Wed Oct 2 17:42:13 UTC 2019

I finally figured out what is happening.   I could use some help fixing the
problem.

The client (behind the virtual Firewall) sends an ARP Request and the
Firewall sends the ARP Request on.   ESX is echoing the Request back to the
Firewall, which is causing the Firewall to drop the ARP Reply when it
arrives.   On rare occasions the server sends the ARP Reply fast enough
that it arrives before the echoed Request and the Reply is passed.

How do I get the firewall to pass the ARP Reply even though it has received
the same Request back?

On Wed, Jul 24, 2019 at 1:17 PM Dan Lists <lists.dan at gmail.com> wrote:

>
> Does anyone have any ideas on how to debug this issue?    I can build a
> custom kernel or use dtrace if necessary.  I'm not familiar with the kernel
> source so I don't really know where to look.
>
> Thanks for any help you can provide.
>
>
> On Mon, Jul 8, 2019 at 12:19 PM Dan Lists <lists.dan at gmail.com> wrote:
>
>>
>> On Mon, Jul 8, 2019 at 11:55 AM Michael Sierchio <kudzu at tenebras.com>
>> wrote:
>>
>>> What's your firewall ruleset look like?  (show, don't tell)
>>>
>>
>> The firewall is off for testing (the machine is only on a private
>> network).
>>
>>  # ipfw list
>> 65535 allow ip from any to any
>>
>>
>>> What does sysctl report on the interfaces and on arp?
>>>
>>>
>> I have not changed any settings.
>>
>> net.link.ether.inet.log_arp_permanent_modify: 1
>>
>> net.link.ether.inet.log_arp_movements: 1
>>
>> net.link.ether.inet.log_arp_wrong_iface: 1
>>
>> net.link.ether.inet.garp_rexmit_count: 0
>>
>> net.link.bridge.allow_llz_overlap: 0
>>
>> net.link.bridge.inherit_mac: 0
>>
>> net.link.bridge.log_stp: 0
>>
>> net.link.bridge.pfil_local_phys: 0
>>
>> net.link.bridge.pfil_member: 1
>>
>> net.link.bridge.pfil_bridge: 1
>>
>> net.link.bridge.ipfw_arp: 0
>> net.link.bridge.pfil_onlyip: 1
>>
>> hw.em.eee_setting: 1
>>
>> hw.em.rx_process_limit: 100
>>
>> hw.em.enable_msix: 1
>>
>> hw.em.sbp: 0
>>
>> hw.em.smart_pwr_down: 0
>>
>> hw.em.txd: 1024
>>
>> hw.em.rxd: 1024
>>
>> hw.em.rx_abs_int_delay: 66
>>
>> hw.em.tx_abs_int_delay: 66
>>
>> hw.em.rx_int_delay: 0
>>
>> hw.em.tx_int_delay: 66
>>
>> hw.em.disable_crc_stripping: 0
>>
>> dev.em.2.wake: 0
>>
>> dev.em.2.mac_stats.tso_ctx_fail: 0
>>
>> dev.em.2.mac_stats.tso_txd: 0
>>
>> dev.em.2.mac_stats.tx_frames_1024_1522: 3
>>
>> dev.em.2.mac_stats.tx_frames_512_1023: 2
>>
>> dev.em.2.mac_stats.tx_frames_256_511: 5675
>>
>> dev.em.2.mac_stats.tx_frames_128_255: 4255
>>
>> dev.em.2.mac_stats.tx_frames_65_127: 35563
>>
>> dev.em.2.mac_stats.tx_frames_64: 918055
>>
>> dev.em.2.mac_stats.mcast_pkts_txd: 0
>>
>> dev.em.2.mac_stats.bcast_pkts_txd: 0
>>
>> dev.em.2.mac_stats.good_pkts_txd: 963553
>>
>> dev.em.2.mac_stats.total_pkts_txd: 963553
>>
>> dev.em.2.mac_stats.good_octets_txd: 60423175
>>
>> dev.em.2.mac_stats.good_octets_recvd: 16906
>>
>> dev.em.2.mac_stats.rx_frames_1024_1522: 1
>>
>> dev.em.2.mac_stats.rx_frames_512_1023: 0
>>
>> dev.em.2.mac_stats.rx_frames_256_511: 0
>>
>> dev.em.2.mac_stats.rx_frames_128_255: 0
>>
>> dev.em.2.mac_stats.rx_frames_65_127: 235
>>
>> dev.em.2.mac_stats.rx_frames_64: 0
>>
>> dev.em.2.mac_stats.mcast_pkts_recvd: 0
>>
>> dev.em.2.mac_stats.bcast_pkts_recvd: 0
>>
>> dev.em.2.mac_stats.good_pkts_recvd: 236
>>
>> dev.em.2.mac_stats.total_pkts_recvd: 236
>>
>> dev.em.2.mac_stats.xoff_txd: 0
>>
>> dev.em.2.mac_stats.xoff_recvd: 0
>>
>> dev.em.2.mac_stats.xon_txd: 0
>>
>> dev.em.2.mac_stats.xon_recvd: 0
>>
>> dev.em.2.mac_stats.coll_ext_errs: 0
>>
>> dev.em.2.mac_stats.alignment_errs: 0
>>
>> dev.em.2.mac_stats.crc_errs: 0
>>
>> dev.em.2.mac_stats.recv_errs: 0
>>
>> dev.em.2.mac_stats.recv_jabber: 0
>>
>> dev.em.2.mac_stats.recv_oversize: 0
>>
>> dev.em.2.mac_stats.recv_fragmented: 0
>>
>> dev.em.2.mac_stats.recv_undersize: 0
>>
>> dev.em.2.mac_stats.recv_no_buff: 0
>>
>> dev.em.2.mac_stats.missed_packets: 0
>>
>> dev.em.2.mac_stats.defer_count: 0
>>
>> dev.em.2.mac_stats.sequence_errors: 0
>>
>> dev.em.2.mac_stats.symbol_errors: 0
>>
>> dev.em.2.mac_stats.collision_count: 0
>>
>> dev.em.2.mac_stats.late_coll: 0
>>
>> dev.em.2.mac_stats.multiple_coll: 0
>>
>> dev.em.2.mac_stats.single_coll: 0
>>
>> dev.em.2.mac_stats.excess_coll: 0
>>
>> dev.em.2.rxd_tail: 235
>>
>> dev.em.2.rxd_head: 236
>>
>> dev.em.2.txd_tail: 127
>>
>> dev.em.2.txd_head: 127
>>
>> dev.em.2.fifo_reset: 0
>>
>> dev.em.2.fifo_workaround: 0
>>
>> dev.em.2.fc_low_water: 45604
>>
>> dev.em.2.fc_high_water: 47104
>>
>> dev.em.2.rx_control: 32794
>>
>> dev.em.2.device_control: 1086325321
>>
>> dev.em.2.watchdog_timeouts: 0
>>
>> dev.em.2.rx_overruns: 0
>>
>> dev.em.2.tx_desc_fail2: 0
>>
>> dev.em.2.tx_desc_fail1: 0
>>
>> dev.em.2.tx_dma_fail: 0
>>
>> dev.em.2.dropped: 0
>>
>> dev.em.2.mbuf_defrag_fail: 0
>>
>> dev.em.2.cluster_alloc_fail: 0
>>
>> dev.em.2.flow_control: 3
>>
>> dev.em.2.rx_processing_limit: 100
>>
>> dev.em.2.itr: 488
>>
>> dev.em.2.tx_abs_int_delay: 66
>>
>> dev.em.2.rx_abs_int_delay: 66
>>
>> dev.em.2.tx_int_delay: 66
>>
>> dev.em.2.rx_int_delay: 0
>>
>> dev.em.2.nvm: -1
>>
>> dev.em.2.%parent: pci2
>>
>> dev.em.2.%pnpinfo: vendor=0x8086 device=0x100f subvendor=0x15ad
>> subdevice=0x0750 class=0x020000
>>
>> dev.em.2.%location: slot=3 function=0 dbsf=pci0:2:3:0
>> handle=\_SB_.PCI0.P2P0.S4F0
>>
>> dev.em.2.%driver: em
>>
>> dev.em.2.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.1.0
>>
>> dev.em.1.wake: 0
>>
>> dev.em.1.mac_stats.tso_ctx_fail: 0
>>
>> dev.em.1.mac_stats.tso_txd: 0
>>
>> dev.em.1.mac_stats.tx_frames_1024_1522: 1
>>
>> dev.em.1.mac_stats.tx_frames_512_1023: 0
>>
>> dev.em.1.mac_stats.tx_frames_256_511: 0
>>
>> dev.em.1.mac_stats.tx_frames_128_255: 0
>>
>> dev.em.1.mac_stats.tx_frames_65_127: 13
>>
>> dev.em.1.mac_stats.tx_frames_64: 222
>>
>> dev.em.1.mac_stats.mcast_pkts_txd: 0
>>
>> dev.em.1.mac_stats.bcast_pkts_txd: 0
>>
>> dev.em.1.mac_stats.good_pkts_txd: 236
>>
>> dev.em.1.mac_stats.total_pkts_txd: 236
>>
>> dev.em.1.mac_stats.good_octets_txd: 15962
>>
>> dev.em.1.mac_stats.good_octets_recvd: 5286574431
>>
>> dev.em.1.mac_stats.rx_frames_1024_1522: 2876152
>>
>> dev.em.1.mac_stats.rx_frames_512_1023: 269810
>>
>> dev.em.1.mac_stats.rx_frames_256_511: 605978
>>
>> dev.em.1.mac_stats.rx_frames_128_255: 1879331
>>
>> dev.em.1.mac_stats.rx_frames_65_127: 3179015
>>
>> dev.em.1.mac_stats.rx_frames_64: 6
>>
>> dev.em.1.mac_stats.mcast_pkts_recvd: 0
>>
>> dev.em.1.mac_stats.bcast_pkts_recvd: 0
>>
>> dev.em.1.mac_stats.good_pkts_recvd: 8809664
>>
>> dev.em.1.mac_stats.total_pkts_recvd: 8816562
>>
>> dev.em.1.mac_stats.xoff_txd: 0
>>
>> dev.em.1.mac_stats.xoff_recvd: 0
>>
>> dev.em.1.mac_stats.xon_txd: 0
>>
>> dev.em.1.mac_stats.xon_recvd: 0
>>
>> dev.em.1.mac_stats.coll_ext_errs: 0
>>
>> dev.em.1.mac_stats.alignment_errs: 0
>>
>> dev.em.1.mac_stats.crc_errs: 0
>>
>> dev.em.1.mac_stats.recv_errs: 0
>>
>> dev.em.1.mac_stats.recv_jabber: 0
>>
>> dev.em.1.mac_stats.recv_oversize: 0
>>
>> dev.em.1.mac_stats.recv_fragmented: 0
>>
>> dev.em.1.mac_stats.recv_undersize: 0
>>
>> dev.em.1.mac_stats.recv_no_buff: 0
>>
>> dev.em.1.mac_stats.missed_packets: 0
>>
>> dev.em.1.mac_stats.defer_count: 0
>>
>> dev.em.1.mac_stats.sequence_errors: 0
>>
>> dev.em.1.mac_stats.symbol_errors: 0
>>
>> dev.em.1.mac_stats.collision_count: 0
>>
>> dev.em.1.mac_stats.late_coll: 0
>>
>> dev.em.1.mac_stats.multiple_coll: 0
>>
>> dev.em.1.mac_stats.single_coll: 0
>>
>> dev.em.1.mac_stats.excess_coll: 0
>>
>> dev.em.1.rxd_tail: 167
>>
>> dev.em.1.rxd_head: 168
>>
>> dev.em.1.txd_tail: 236
>>
>> dev.em.1.txd_head: 236
>>
>> dev.em.1.fifo_reset: 0
>>
>> dev.em.1.fifo_workaround: 0
>>
>> dev.em.1.fc_low_water: 45604
>>
>> dev.em.1.fc_high_water: 47104
>>
>> dev.em.1.rx_control: 32794
>>
>> dev.em.1.device_control: 1086325321
>>
>> dev.em.1.watchdog_timeouts: 0
>>
>> dev.em.1.rx_overruns: 0
>>
>> dev.em.1.tx_desc_fail2: 0
>>
>> dev.em.1.tx_desc_fail1: 0
>>
>> dev.em.1.tx_dma_fail: 0
>>
>> dev.em.1.dropped: 0
>>
>> dev.em.1.mbuf_defrag_fail: 0
>>
>> dev.em.1.cluster_alloc_fail: 0
>>
>> dev.em.1.flow_control: 3
>>
>> dev.em.1.rx_processing_limit: 100
>>
>> dev.em.1.itr: 488
>>
>> dev.em.1.tx_abs_int_delay: 66
>>
>> dev.em.1.rx_abs_int_delay: 66
>>
>> dev.em.1.tx_int_delay: 66
>>
>> dev.em.1.rx_int_delay: 0
>>
>> dev.em.1.nvm: -1
>>
>> dev.em.1.%parent: pci2
>>
>> dev.em.1.%pnpinfo: vendor=0x8086 device=0x100f subvendor=0x15ad
>> subdevice=0x0750 class=0x020000
>>
>> dev.em.1.%location: slot=2 function=0 dbsf=pci0:2:2:0
>> handle=\_SB_.PCI0.P2P0.S3F0
>>
>> dev.em.1.%driver: em
>>
>> dev.em.1.%desc: Intel(R) PRO/1000 Legacy Network Connection 1.1.0
>>
>>
>>>
>>> On Mon, Jul 8, 2019 at 9:15 AM Dan Lists <lists.dan at gmail.com> wrote:
>>>
>>> > I have a server running FreeBSD 11.2 that I am wanting to use as a
>>> bridged
>>> > firewall.  I have it set up and it mostly works.   The problem is that
>>> ARP
>>> > replies are not being forwarded from the outside interface to the
>>> inside
>>> > interface.   It appears to be working in the other direction.  I see
>>> the
>>> > ARP request go out on the outside interface and the reply arrives back
>>> at
>>> > the outside interface.   The ARP reply is never getting to the bridge
>>> or to
>>> > the inside interface.
>>> >
>>> > The firewall server and the device behind it are in ESX.   I think I've
>>> > worked all the ESX issues out.  When I manually add an ARP entry
>>> everything
>>> > works.   I've done this before with a physical server running FreeBSD
>>> 8.4
>>> > and it works as expected.   The differences are physical vs virtual,
>>> and
>>> > 8.4 vs 11.2.
>>> >
>>> > I'm at a loss as to why it is not working.   I've searched the web and
>>> > found noting.  If anyone could offer suggestions on how to fix this or
>>> > begin to debug it I would greatly appreciate it.
>>> >
>>> > Thanks,
>>> >
>>> > Dan
>>> > _______________________________________________
>>> > freebsd-net at freebsd.org mailing list
>>> > https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> > To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>> >
>>>
>>>
>>> --
>>>
>>> "Well," Brahmā said, "even after ten thousand explanations, a fool is no
>>> wiser, but an intelligent person requires only two thousand five
>>> hundred."
>>>
>>> - The Mahābhārata
>>> _______________________________________________
>>> freebsd-net at freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>
>>