increasing transmit speeds in WAN setting?

[Moses Leslie]

Hi Ted,

While I don't totally discount that possibility, I really don't think
that's the case.  We have over 500 servers, most of them running FreeBSD,
and we've seen this happen in multiple cases on different hardware. When
it's linux, exact same hardware, exact same cables, this doesn't happen.

It's an intel card gbit card, using the em driver.  They're uplinked to
Cisco 2948-getx switches, which are uplinked to 65xx's, which then go to
12xxx borders.  There aren't any collision errors on the port at all:

24 totalCollisionCount        = 0
25 lateCollisionCount         = 0
26 singleCollisionFrames      = 0
27 multipleCollisionFrames    = 0
28 excessiveCollisionFrames   = 0

and no real errors to speak of, period.

The port is auto, since it needs to be to get gbit.  All of the non-gbit
servers we have are forced 100/full, all cisco switches, all intel 100/pro
(fxp) drivers, they all show this same problem.

If the server is a 4.9 server, I can get ~400KB/s.  If it's 6.1, ~300KB/s.
Linux 2.6, ~650KB/s, which is about what I'd expect given the latency and
the default settings.  All on the same hardware, same switches, same
cables.

The only error that increments at all is txQueueNotAvailable, which is to
be expected as the BDP is figured out.

I'm pretty sure that FreeBSD is throttling itself back when it shouldn't
be.

Thanks for the reply,

Moses

On Wed, 18 Oct 2006, Ted Mittelstaedt wrote:

> Hi Moses,
>
> I know your not going to believe me but you are running into a
> driver bug of some kind.  If you have a really high quality ethernet
> switch with full management in it you can probably see it - login to
> the switch and look at the port statistics. Cisco routers are designed
> to sense for this and you will see it in their logs, they will issue the
> error message "late collissions" or any decent hardware network
> sniffer will show it.
>
> The most common problem is the switch and network card aren't
> properly negotiating duplex.  Another area is flow control on full
> duplex being messed up, this is particularly critical on gigabit E.
>
> The reason your getting good throughput on local connections is
> that the layer 1 is simply continuing to retransmit until the packet
> goes through, and the retransmissions are happening so fast that
> you don't realize it.  That is also why latency is so heavily affecting
> it.
>
> You can try several things.  First, temporarily try switching
> over to a 10/100 card like an Intel EtherExpress Pro/100
> if you have a PCI slot in the server.  If that works then your going
> to have to try replacing your switch.  If you have a really good
> switch you can try hard coding it's ports speed and duplex and
> try the same on the server, and see if that does anything.
>
> You also should be aware that many of the smaller and cheaper
> gigabit switches do not have the ability to take sustained
> gigabit ethernet speeds with back-to-back packets, their
> internal processors aren't fast enough.  Once more, this is
> a problem that won't show up on a local connection since the
> retransmissions are so fast.
>
> Ted
>
> ----- Original Message -----
> From: "Moses Leslie" <marmoset at malformed.org>
> To: <freebsd-questions at freebsd.org>
> Sent: Wednesday, October 18, 2006 10:31 PM
> Subject: increasing transmit speeds in WAN setting?
>
>
> > Hi,
> >
> > We're running 6.1-R, and are having difficulty getting decent speeds as
> > latency increases.  The server is connected via gbit copper, and is gbit
> > or better to the internet (depending on the path).
> >
> > For everything local, we're able to get what you'd expect (300+MBit
> > without really any tuning).  However, when the latency is 60-80ms (IE
> > across the US), we're unable to get better than around 300KB/s.
> >
> > It appears to be possibly related to the tcp.inflight stuff, but disabling
> > it or messing with some of the related sysctls doesn't appear to help
> > much.  Downloads often start quickly, but are then throttled back down to
> > 300KB/s within 10 seconds or so.  We've changed the hz (100 to 10000), the
> > net.inet.tcp.sendspace, kern.ipc.maxsockbuf, and tried different
> > variations on the inflight tunables, but nothing has made a positive
> > difference of more than ~20KB/s at best.
> >
> > If the server is running linux (2.6 kernel with default TCP settings), we
> > can get much better speeds, 600-1000KB/s easily.  If we were going for
> > time/distance records, we would try changing around tcp settings on the
> > client, but we're trying to maximize performance for standard surfers who
> > wouldn't know how to do that, so we're looking for anything that is server
> > side only.
> >
> > We've been searching high and low for any tuning ideas but aren't able to
> > find anything that's made a difference.  From looking at how the
> > congestion stuff works in the source, it appears that something like:
> >
> > http://www.sigusr1.org/weblog/index.php?/categories/6-Hacks
> >
> > might be happening here, but we're kind of stabbing in the dark.
> >
> > Does anyone have any tuning ideas for 6.1 in a WAN setting?
> >
> > Thanks,
> >
> > Moses
> >
> >
> >
> >
> >
> > _______________________________________________
> > freebsd-questions at freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to
> "freebsd-questions-unsubscribe at freebsd.org"
> >
>