bce(4) - com_no_buffers (Again)

[Pyun YongHyeon]

On Mon, Sep 13, 2010 at 02:07:58PM -0500, Tom Judge wrote:
> On 09/13/2010 01:48 PM, Pyun YongHyeon wrote:
> > On Mon, Sep 13, 2010 at 10:04:25AM -0500, Tom Judge wrote:
> >   
> >>
> <SNIP/>
> >> Does this mean that these cards are going to perform badly? This is was
> >> what I gathered from the previous thread.
> >>
> >>     
> > I mean there are still many rooms to be done in driver for better
> > performance. bce(4) controllers are one of best controllers for
> > servers and driver didn't take full advantage of it.
> >
> >   
> 
> So far our experiences with bce(4) on FreeBSD have been very
> disappointing.  Starting with when Dell switched to bce(4) based NIC's
> (around the time 6.2 was released and with the introduction of the Power
> Edge X9XX hardware) we have always had problems with the driver in every
> release we have used: 6.2, 7.0 and 7.1.  Luckily David has been helpful
> and helped us fix the issues.
> 
> <SNIP/>
> >   
> >> Without BCE_JUMBO_HDRSPLIT then we see no errors.  With it we see number
> >> of errors, however the rate seems to be reduced compaired to the
> >> previous version of the driver.
> >>
> >>     
> > It seems there are issues in header splitting and it was disabled
> > by default. Header splitting reduces packet processing overhead in
> > upper layer so it's normal to see better performance with header
> > splitting.
> >   
> 
> The reason that we have had header splitting enabled in the past is that
> historically there have been issues with memory fragmentation when using
> 8k jumbo frames (resulting in 9k mbuf's).
> 

Yes, if you use jumbo frames, header splitting would help to reduce
memory fragmentation as header splitting wouldn't allocate jumbo
clusters.

> I have a kernel with the following configuration in testing right now:
> 
> * Flow control enabled.
> * Jumbo header splitting turned off.
> 
> 
> Is there any way that we can fix flow control with jumbo header
> splitting turned on?
> 

Flow control has nothing to do with header splitting(i.e. flow
control is always enabled). 

> Thanks
> 
> Tom
> 
> PS. The following test was more than enough to trigger buffer shortages
> with header splitting on:
> 
> ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done
> ) &
> ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done
> ) &
> ( while true; do ldapsearch -h ldap-server1 -b "ou=Some,o=Base" dn; done
> ) &
> 
> The search in question returned about 1700 entries.
> 

I can trigger this kind of buffer shortage with benchmark tools.
Actually fixing header splitting is on my TODO list as well as
other things but I don't know how long it would take.