em0 performance subpar

[Adam Stylinski]

---------- Forwarded message ----------
From: Adam Stylinski <kungfujesus06 at gmail.com>
Date: Thu, Apr 28, 2011 at 8:38 AM
Subject: Re: em0 performance subpar
To: Eugene Grosbein <egrosbein at rdtc.ru>


On Thu, Apr 28, 2011 at 05:51:58PM +0700, Eugene Grosbein wrote:
> On 28.04.2011 14:29, Adam Stylinski wrote:
> > Hello,
> >
> > I have an intel gigabit network adapter (the 1000 GT w/chipset 82541PI)
which performs poorly in Freebsd compared to the same card in Linux.  I've
tried this card in two different freebsd boxes and for whatever reason I get
poor transmit performance.  I've done all of the tweaking specified in just
about every guide out there (the usual TCP window scaling, larger
nmbclusters, delayed acks, etc) and still I get only around 600mbps.  I'm
using jumbo frames, with an MTU of 9000.  I'm testing this with iperf.
 While I realize that this may not be the most realistic test, linux hosts
with the same card can achieve 995Mbit/s to another host running this.  When
the Freebsd box is the server, Linux hosts can transmit to it at around 800
something Mbit/s.  I've increased the transmit descriptors as specified in
the if_em man page, and while that gave me 20 or 30 more mbit/s, my transmit
performance is still below normal.
> >
> > sysctl stats report that the card is trigger a lot of tx_desc_fail2:
> >     dev.em.0.tx_desc_fail2: 3431
> >
> > Looking at a comment in the source code this indicates that the card was
not able to obtain enough transmit descriptors (but I've given the card the
maximum of 4096 in my loader.conf tunable).  Is this a bug or a performance
regression of some kind?  Does anybody have a fix for this?  I tried another
card with the same chip in a different box on 8-STABLE to no avail (the box
I'm trying to improve performance on is on 8.2-RELEASE-p1).
> >
> > Anybody manage to make this card push above 600mbps in ideal network
benchmarks?  Any help would be gladly appreciated.
>
> Please show output of "vmstat -z|egrep 'ITEM|mbuf'" command after the
test.
>
> Eugene Grosbein

ITEM                     SIZE     LIMIT      USED      FREE  REQUESTS
 FAILURES
mbuf_packet:              256,        0,     4096,      896,   531481,
 0
mbuf:                     256,        0,     1026,    10497,  3882824,
 0
mbuf_cluster:            2048,    65768,     4992,     2998,   105908,
 0
mbuf_jumbo_page:         4096,    12800,        0,     4462,  1261679,
 0
mbuf_jumbo_9k:           9216,     6400,        0,        0,        0,
 0
mbuf_jumbo_16k:         16384,     3200,        0,        0,        0,
 0
mbuf_ext_refcnt:            4,        0,        0,      168,        1,
 0

I'm willing to try compiling the module with EM_MULTIQUEUE on the test box
to see if it makes a difference, but I read it can have issues with UDP
traffic.
--
Adam Stylinski
PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub
Blog: http://technicallyliving.blogspot.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20110428/85054911/attachment.pgp