Errors using span interface on if_bridge(4)

[hiren panchasara]

I rebuilt the kernel with following change to check failure:

Index: if_bridge.c
===================================================================
--- if_bridge.c (revision 260789)
+++ if_bridge.c (working copy)
@@ -2536,6 +2536,7 @@
        struct bridge_iflist *bif;
        struct ifnet *dst_if;
        struct mbuf *mc;
+       int error = 0;

        if (LIST_EMPTY(&sc->sc_spanlist))
                return;
@@ -2552,7 +2553,9 @@
                        continue;
                }

-               bridge_enqueue(sc, dst_if, mc);
+               error = bridge_enqueue(sc, dst_if, mc);
+               if (error)
+                       printf("%s: bridge_enqueue failed\n", __func__);
        }
 }

After this change and reboot, I see packets on ix3 without those
bad-len errors seen before. (I also want to carefully inspect packets
to make sure nothing else is going wrong)
I never saw "bridge_enqueue failed" error ever. So its failing
somewhere else in the stack.

ix1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
        options=8403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
        ether 38:ea:a7:8b:af:c4
        inet6 fe80::3aea:a7ff:fe8b:afc4%ix1 prefixlen 64 scopeid 0x6
        inet 10.73.149.91 netmask 0xffffff00 broadcast 10.73.149.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
        status: active
ix2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
        options=8407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
        ether 90:e2:ba:30:73:40
        inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
        status: active
ix3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 1500
        options=8407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
        ether 90:e2:ba:30:73:41
        inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-Twinax <full-duplex>)
        status: active
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:a1:25:9a:8f:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: ix1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: ix2 flags=8<SPAN>
                ifmaxaddr 0 port 7 priority 128 path cost 2000

Interestingly I still see:

-bash-4.2$ sysctl -a dev.ix | grep total_pkts_rcvd
dev.ix.0.mac_stats.total_pkts_rcvd: 638917
dev.ix.1.mac_stats.total_pkts_rcvd: 3587878  <--- where I send traffic via iperf
dev.ix.2.mac_stats.total_pkts_rcvd: 1
dev.ix.3.mac_stats.total_pkts_rcvd: 3555298  <--- where I snoop traffic.

-bash-4.2$ sysctl -a | grep mac_stats.checksum_errs
dev.ix.0.mac_stats.checksum_errs: 0
dev.ix.1.mac_stats.checksum_errs: 0
dev.ix.2.mac_stats.checksum_errs: 0
dev.ix.3.mac_stats.checksum_errs: 3371119  <-- most of them fail checksum

I also noticed:

dev.ix.3.queue4.lro_queued: 50223
dev.ix.3.queue4.lro_flushed: 42632
dev.ix.3.queue5.lro_queued: 25356
dev.ix.3.queue5.lro_flushed: 25249

I disabled tso and lro both and I stopped seeing these number
increase. But  the general behavior remained the same.

cheers,
Hiren