dev.bce.X.com_no_buffers increasing and packet loss
Pyun YongHyeon
pyunyh at gmail.com
Wed Mar 10 23:02:32 UTC 2010
On Wed, Mar 10, 2010 at 02:45:47PM -0800, David Christensen wrote:
>
> > I successfully reproduced the issue with netperf on BCM5709.
> > You can use UDP frame size 1 to trigger the issue.
> >
> > > Changing the high level design of bce_rx_intr() and
> > > bce_rx_fill_chain() slightly to post a new buffer as each frame is
> > > passed to the OS would likely avoid these gaps during
> > bursts of small
> > > frames but I'm not sure whether they'll have a negative
> > impact on the
> > > more common case of streams of MTU sized frames. I've
> > considered this
> > > in the past but never coded the change and tested the resulting
> > > performance.
> > >
> >
> > I guess this may slightly increase performance with additional
> > bus_dma(9) overheads but I think one of reason of dropping
> > frames under heavy UDP frames may come from lack of free RX
> > descriptors.
> > Because bce(4) just uses a single RX ring so the number of
> > available RX buffers would be 512. However it seems it's not
> > possible to increase the number of RX buffers per RX ring so
> > the next possible approach would be switching to use multiple
> > RX rings with RSS. Even though FreeBSD does not dynamically
> > adjust loads among CPUs I guess using RSS would be the way to go.
>
> The bce(4) hardware supports a linked list of pages for RX
> buffer descriptors. The stock build supports 2 pages (RX_PAGES)
> with a total of 511 BD's per page. The hardware can support a
> maximum of 64K BD's but that would be an unnecessarily large
> amount of mbufs for an infrequent problem.
>
Thanks for the info. I guess 2048 or 4096 BDs would be necessary to
get satisfactory Rx performance. I'll have to experiment this.
> The middle road would probably involve changing RX_PAGES from a
> #define to a sysctl variable to allow tuning for specific
> environments along with a change in bce_rx_intr() to fill the
> ring after all frames have been processed or when more than
> 256 BDs have been consumed, whichever comes first.
>
> RSS would be great as well though it wouldn't make a dent in
> this case since RSS is only supported for TCP, not UDP.
>
Even though UDP is not supported in RSS, RSS can handle IP. This
wouldn't distribute UDP load coming from a single host but if
source IP address is different it may help, I guess.
> Dave
More information about the freebsd-current
mailing list