cvs commit: src/sys/dev/bce if_bce.c if_bcefw.h if_bcereg.h

[David Christensen]

> On Thu, Feb 21, 2008 at 5:46 PM, David Christensen
> <davidch at freebsd.org> wrote:
> >   Modified files:
> >     sys/dev/bce          if_bce.c if_bcefw.h if_bcereg.h
> >   Log:
> >   MFC after:      4 weeks
> >
> >   - Added loose RX MTU functionality to allow frames larger
> than 1500 bytes
> >     to be accepted even though the interface MTU is set to 1500.
> >   - Implemented new TCP header splitting/jumbo frame
> support which uses
> >     two chains for receive traffic rather than the original
> single recevie
> >     chain.
> >   - Added additional debug support code.
> >
> >   Revision  Changes      Path
> >   1.36      +1559 -675   src/sys/dev/bce/if_bce.c
> >   1.5       +6179 -4850  src/sys/dev/bce/if_bcefw.h
> >   1.17      +264 -55     src/sys/dev/bce/if_bcereg.h
>
> This has been devastating on the freebsd.org cluster.
>
> Attached are three test runs.  I've done a cold reboot, then 'cd
> /usr/src/sys' and doing a 'cvs -Rq update' where the CVSROOT is over
> nfs.
>
> First, the old driver:
> svn# time cvs -Rq up
> 0.890u 4.577s 1:14.48 7.3%      669+2315k 7379+0io 10094pf+0w
>
> Now, the same test again, but with this change included in the kernel:
> svn# time cvs -Rq up
> 0.940u 359.906s 7:01.04 85.7%   648+2242k 7365+0io 10082pf+0w
>
> Note the massive increase (nearly 100 times increase) in system time,
> and the almost 7-fold increase in wall clock time.
>
> Turning on promisc mode helps a lot, but doesn't solve it.  (This was
> found when ps@ was using tcpdump to try and figure out what the
> problem was)

The change is needed to update the FreeBSD driver so that it can
continue using production firmware for the controllers.  The previous
firmware was specific to FreeBSD and was not being maintained.

I didn't see any performance issues running with netperf.  Is the NFS
traffic UDP or TCP?  What's the MTU in use?  How much system memory is
available?

If this is a performance problem then the first place I would look is
in the definitions for rx_bd_mbuf_alloc_size and pg_bd_mbuf_alloc_size.
The older version of the driver would use multiple 2KB buffers
(MCLBYTES in size) from a single chain when building a packet so you
would typically have a single mbuf cluster passed to the stack.  The
new firmware uses two chains, each of which may be a different size.
The current implementation will use MHLEN bytes for the rx chain and
MCLBYTES for the pg chain.  When a packet is received the hardware will
place as much data as possible into a single mbuf in the rx chain,
then place any remaining data into one or more mbufs in the pg chain.
The driver will then stitch together the mbufs before passing them up
the stack.  This process is supposed to improve performance for TCP
because the TCP payload will be split from the TCP header and should
be quicker to access.

A quick test would be to set rx_bd_mbuf_alloc_size to MCLBYTES, which
should for the most part duplicate the older behavior.  The driver
will still allocate more mbufs which might be a problem if system
memory is already low.  Is anyone else aware of a driver that does
TCP header splitting?  It's typically on the TX side to see a packet
with two or three mbufs in a chain but I suspect it's less typical
on the RX side which could be part of the problem.

>
> Here's the same test, with the new driver, and promisc mode on:
> svn# ifconfig bce0 promisc
> svn# time cvs -Rq up
> 0.967u 50.919s 2:13.97 38.7%    650+2250k 7379+0io 10094pf+0w
>
> It is better.. Only double the wall clock time, but still over 10
> times as much system time.
>

It's not clear to me why promiscuous mode would make a difference
here as that should only affect which packets are accepted by the
MAC.  Is there any teaming or VLANs used in your configuration?
The RX MTU settings shouldn't be affected by promiscuous mode.

>
> So please, don't MFC until this is solved..
>

I haven't yet as I've received reports from a few other people that
they're having problems, though they're functional problems and not
performance issues.