increasing transmit speeds in WAN setting?
tedm at toybox.placo.com
Fri Oct 20 06:43:32 UTC 2006
----- Original Message -----
From: "Moses Leslie" <marmoset at malformed.org>
To: "Ted Mittelstaedt" <tedm at toybox.placo.com>
Cc: <freebsd-questions at freebsd.org>
Sent: Thursday, October 19, 2006 1:33 AM
Subject: Re: increasing transmit speeds in WAN setting?
> Hi Ted,
> While I don't totally discount that possibility, I really don't think
> that's the case.
I told you that you wouldn't believe me.
> We have over 500 servers, most of them running FreeBSD,
> and we've seen this happen in multiple cases on different hardware.
Except all of them gigabit cards, right? So much for "different hardware"
> it's linux, exact same hardware, exact same cables, this doesn't happen.
> It's an intel card gbit card, using the em driver. They're uplinked to
> Cisco 2948-getx switches, which are uplinked to 65xx's, which then go to
> 12xxx borders. There aren't any collision errors on the port at all:
> 24 totalCollisionCount = 0
> 25 lateCollisionCount = 0
> 26 singleCollisionFrames = 0
> 27 multipleCollisionFrames = 0
> 28 excessiveCollisionFrames = 0
> and no real errors to speak of, period.
> The port is auto, since it needs to be to get gbit. All of the non-gbit
> servers we have are forced 100/full, all cisco switches, all intel 100/pro
> (fxp) drivers, they all show this same problem.
Well right there you are doing things wrong. You should always set
ethernet cards to auto. The only time you ever force 100/full or force
anything, speed/duplex, is when your plugged into a hub that does NOT
autoswitch. There's very few of them around that are 100base T, but
there are some, and there's a lot more 10baseT stuff that wasn't
Any halfway decent 100baseT hub will support nway autonegotiation and
when you hard-code post speeds you will cause drops and speed loss.
But, please don't take my word for it since you seem to like disbelieving
me, just try it out yourself.
Go to your fxp servers, login to your switches, set the switch port to the
server to autonegotiation, on the server remove all the media options in
shut down the server (you must power it down for the ports to switch
into autonegotiation) and bring it up and you will see both sides negotiate
to 100base T full, and a lot of your problems in throughput will disappear.
Both the switch and the servermust be set to autonegotiation. If they
don't autonegotiate to 100baseTfull, then you have a cable problem, simple
I've been doing ethernet since the late 80's and doing it professionally
for a decade, and I've seen and use more different types of ethernet in my
life than you will ever see in the rest of your career.
The idea that your supposed to override the autonegotiation and hard
code stuff originated from network admins who plugged early 100baseT
stuff together then couldn't figure out why it didn't autonegotiate to
full. What they didn't realze is that the cabling they were using - CAT-3
or CAT-5 that had been incorrectly terminated with the wrong connectors, or
wrong plugs, or wrong wiring pattern, or bad crimps because they were using
stranded plugs on solid core wire, or some other such thing, what the real
culprit, and the autonegotiation chips were in fact detecting the problem
and trying to protect the network.
Unfortunately, 90% of network admins out there don't know the first thing
about layer-1, they assume the wiring contractors handle all that. The
contractors by contrast are mostly minimum-wage goobers who's heads
are filled with a lot of rediculous nonsense about how Ethernet really
> If the server is a 4.9 server, I can get ~400KB/s. If it's 6.1, ~300KB/s.
> Linux 2.6, ~650KB/s, which is about what I'd expect given the latency and
> the default settings. All on the same hardware, same switches, same
The Linux device drivers are simply different than the FreeBSD drivers.
I don't know how much more I can tell you this over and over. The em
driver has got some problems, granted. But, this has absolutely nothing
to do with the FreeBSD version or the TCP/IP stack.
Until you do what I told you to do and properly setup and test under
fxp0, I am just not going to waste my time on this anymore. I will
leave you with a printout of a test run on a new mailserver I'm building up
right now, in fact, using an fxp card, to prove it's a not a stack problem.
You can choose to believe it or you can choose to continue wasting your
time chasing ghosts in the TP stack when the problem is the driver:
$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/ls-lR.gz
ls-lR.gz 100% of 18 MB 1057 kBps
$ ping ftp.freebsd.org
PING ftp.freebsd.org (22.214.171.124): 56 data bytes
36 bytes from ge2-16.1000M.d5.opa.tdk.net (126.96.36.199): Communication
prohibited by filter
Vr HL TOS Len ID Flg off TTL Pro cks Src Dst
4 5 00 5400 82f2 0 0000 33 01 6e38 188.8.131.52 184.108.40.206
--- ftp.freebsd.org ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
$ ping 220.127.116.11
PING 18.104.22.168 (22.214.171.124): 56 data bytes
64 bytes from 126.96.36.199: icmp_seq=0 ttl=242 time=171.189 ms
64 bytes from 188.8.131.52: icmp_seq=1 ttl=242 time=171.470 ms
64 bytes from 184.108.40.206: icmp_seq=2 ttl=242 time=171.185 ms
64 bytes from 220.127.116.11: icmp_seq=3 ttl=242 time=171.179 ms
--- 18.104.22.168 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 171.179/171.256/171.470/0.124 ms
$ fetch ftp://ftp.freebsd.org/pub/FreeBSD/ls-lR.gz
ls-lR.gz 100% of 18 MB 1040 kBps
Thu Oct 19 23:35:13 PDT 2006
$ uname -a
FreeBSD mail.seasurf.net 6.1-RELEASE FreeBSD 6.1-RELEASE #0: Wed Sep 20
19:38:01 PDT 2006
tedm at mail.seasurf.net:/usr/src/sys/i386/compile/SEAMAIL i386
$ ifconfig -a
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
inet6 fe80::250:8bff:fee0:4f03%fxp0 prefixlen 64 scopeid 0x1
inet 22.214.171.124 netmask 0xffffffe0 broadcast 126.96.36.199
media: Ethernet autoselect (100baseTX <full-duplex>)
More information about the freebsd-questions