bce(4) - com_no_buffers (Again)

Thu Sep 9 20:58:41 UTC 2010

Hi,

I am just following up on the thread from March (I think) about this issue.

We are seeing this issue on a number of systems running 7.1. 

The systems in question are all Dell:

* R710 R610 R410
* PE2950

The latter do not show the issue as much as the R series systems.

The cards in one of the R610's that I am testing with are:

bce0 at pci0:1:0:0:        class=0x020000 card=0x02361028 chip=0x163914e4
rev=0x20 hdr=0x00
    vendor     = 'Broadcom Corporation'
    device     = 'NetXtreme II BCM5709 Gigabit Ethernet'
    class      = network
    subclass   = ethernet

They are connected to Dell PowerConnect 5424 switches.

uname -a:
FreeBSD bandor.chi-dc.mintel.ad 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4
#3: Wed Sep  8 08:19:03 UTC 2010    
tj at dev-tj-7-1-amd64.chicago.mintel.ad:/usr/obj/usr/src/sys/MINTELv10  amd64

We are also using 8192 byte jumbo frames, if_lagg and if_vlan in the
configuration (the nics are in promisc as we are currently capturing
netflow data on another vlan for diagnostic purposes. ):

tj at bandor '20:51:17' '~'
> $ ifconfig bce0
bce0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 8192

options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
        ether 00:21:9b:95:7a:b8
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
        lagg: laggdev lagg0
tj at bandor '20:51:22' '~'
> $ ifconfig bce1
bce1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 8192

options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
        ether 00:21:9b:95:7a:b8
        media: Ethernet autoselect (1000baseTX <full-duplex>)
        status: active
        lagg: laggdev lagg0
tj at bandor '20:51:35' '~'
> $ ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 8192

options=400bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
        ether 00:21:9b:95:7a:b8
        media: Ethernet autoselect
        status: active
        laggproto failover
        laggport: bce1 flags=0<>
        laggport: bce0 flags=5<MASTER,ACTIVE>
tj at bandor '20:51:40' '~'
> $ ifconfig vlan2
vlan2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric
0 mtu 8192
        options=3<RXCSUM,TXCSUM>
        ether 00:21:9b:95:7a:b8
        inet 172.30.XX.XX netmask 0xfffffe00 broadcast 172.30.XX.XX
        media: Ethernet autoselect
        status: active
        vlan: 2 parent interface: lagg0

I have updated the bce driver and the Broadcomm MII driver to the
version from stable/7 and am still seeing the issue.

This morning I did a test with increasing the RX_PAGES to 8 but the
system just hung starting the network.  The route command got stuck in a
zone state (Sorry can't remember exactly which).

The real question is, how do we go about increasing the number of RX
BDs? I guess we have to bump more that just RX_PAGES...

The cause for us, from what we can see, is the openldap server sending
large group search results back to nss_ldap or pam_ldap.  When it does
this it seems to send each of the 600 results in its own TCP segment
creating a small packet storm (600*~100byte PDU's) at the destination
host.  The kernel then retransmits 2 blocks of 100 results each after
SACK kicks in for the data that was dropped by the NIC.

Thanks in advance

Tom

tj at bandor '20:57:33' '~'
> $ sysctl -a dev.bce.0
dev.bce.0.%desc: Broadcom NetXtreme II BCM5709 1000Base-T (C0)
dev.bce.0.%driver: bce
dev.bce.0.%location: slot=0 function=0
dev.bce.0.%pnpinfo: vendor=0x14e4 device=0x1639 subvendor=0x1028
subdevice=0x0236 class=0x020000
dev.bce.0.%parent: pci1
dev.bce.0.l2fhdr_error_count: 0
dev.bce.0.mbuf_alloc_failed_count: 0
dev.bce.0.mbuf_frag_count: 0
dev.bce.0.dma_map_addr_rx_failed_count: 0
dev.bce.0.dma_map_addr_tx_failed_count: 0
dev.bce.0.unexpected_attention_count: 0
dev.bce.0.stat_IfHcInOctets: 439779802
dev.bce.0.stat_IfHCInBadOctets: 0
dev.bce.0.stat_IfHCOutOctets: 108341440
dev.bce.0.stat_IfHCOutBadOctets: 0
dev.bce.0.stat_IfHCInUcastPkts: 2341369
dev.bce.0.stat_IfHCInMulticastPkts: 26065
dev.bce.0.stat_IfHCInBroadcastPkts: 9191
dev.bce.0.stat_IfHCOutUcastPkts: 1230052
dev.bce.0.stat_IfHCOutMulticastPkts: 2870
dev.bce.0.stat_IfHCOutBroadcastPkts: 45
dev.bce.0.stat_emac_tx_stat_dot3statsinternalmactransmiterrors: 0
dev.bce.0.stat_Dot3StatsCarrierSenseErrors: 0
dev.bce.0.stat_Dot3StatsFCSErrors: 0
dev.bce.0.stat_Dot3StatsAlignmentErrors: 0
dev.bce.0.stat_Dot3StatsSingleCollisionFrames: 0
dev.bce.0.stat_Dot3StatsMultipleCollisionFrames: 0
dev.bce.0.stat_Dot3StatsDeferredTransmissions: 0
dev.bce.0.stat_Dot3StatsExcessiveCollisions: 0
dev.bce.0.stat_Dot3StatsLateCollisions: 0
dev.bce.0.stat_EtherStatsCollisions: 0
dev.bce.0.stat_EtherStatsFragments: 0
dev.bce.0.stat_EtherStatsJabbers: 0
dev.bce.0.stat_EtherStatsUndersizePkts: 0
dev.bce.0.stat_EtherStatsOversizePkts: 0
dev.bce.0.stat_EtherStatsPktsRx64Octets: 3381
dev.bce.0.stat_EtherStatsPktsRx65Octetsto127Octets: 98883
dev.bce.0.stat_EtherStatsPktsRx128Octetsto255Octets: 2255959
dev.bce.0.stat_EtherStatsPktsRx256Octetsto511Octets: 12508
dev.bce.0.stat_EtherStatsPktsRx512Octetsto1023Octets: 4247
dev.bce.0.stat_EtherStatsPktsRx1024Octetsto1522Octets: 522
dev.bce.0.stat_EtherStatsPktsRx1523Octetsto9022Octets: 1125
dev.bce.0.stat_EtherStatsPktsTx64Octets: 496
dev.bce.0.stat_EtherStatsPktsTx65Octetsto127Octets: 1176041
dev.bce.0.stat_EtherStatsPktsTx128Octetsto255Octets: 29079
dev.bce.0.stat_EtherStatsPktsTx256Octetsto511Octets: 2933
dev.bce.0.stat_EtherStatsPktsTx512Octetsto1023Octets: 23898
dev.bce.0.stat_EtherStatsPktsTx1024Octetsto1522Octets: 234
dev.bce.0.stat_EtherStatsPktsTx1523Octetsto9022Octets: 286
dev.bce.0.stat_XonPauseFramesReceived: 0
dev.bce.0.stat_XoffPauseFramesReceived: 0
dev.bce.0.stat_OutXonSent: 0
dev.bce.0.stat_OutXoffSent: 0
dev.bce.0.stat_FlowControlDone: 0
dev.bce.0.stat_MacControlFramesReceived: 0
dev.bce.0.stat_XoffStateEntered: 0
dev.bce.0.stat_IfInFramesL2FilterDiscards: 0
dev.bce.0.stat_IfInRuleCheckerDiscards: 0
dev.bce.0.stat_IfInFTQDiscards: 0
dev.bce.0.stat_IfInMBUFDiscards: 0
dev.bce.0.stat_IfInRuleCheckerP4Hit: 35256
dev.bce.0.stat_CatchupInRuleCheckerDiscards: 0
dev.bce.0.stat_CatchupInFTQDiscards: 0
dev.bce.0.stat_CatchupInMBUFDiscards: 0
dev.bce.0.stat_CatchupInRuleCheckerP4Hit: 0
dev.bce.0.com_no_buffers: 13021

-- 
TJU13-ARIN