tcp_output starving -- is due to mbuf get delay?

[Terry Lambert]

Borje Josefsson wrote:
> > It's interesting - I'm only getting about 320mb/s.. I must be hitting a
> > similar problem.  I'm not nearly as adept at hacking code to find bugs
> > though. :(
> 
> 320 Mbit/sec seems familiar, this was what I got when I first tried on a
> system with "traditional" PCI bus. Changing the OS to NetBSD on that box
> bumped that to 525 Mbit/sec. You need wide PCI (or preferrably PCI-X for
> this).

32bit x 33Mhz = 1.03Gbit/S, burst rate
32bit x 66Mhz = 2.06Gbit/S, burst rate
64bit x 66MHz = 4.12Gbit/S, burst rate

So it's entirely possible to keep up with 2 1Gbit ethernet cards in
standard 64bit PCI bus slots, no problem, without running into bus
limitations.

PCI-X gets you to 8Gbit/S; you don't need PCI-X for Gbit, or even
2Gbit.


> What happens in that case for me is that I run out of CPU resources. Try
> running "top" in one window and "netstat 1" in another while bashing the
> net with ttcp.

This is incredibly bizarre.  It's very hard to saturate the CPU
at only 1Gbit: in all cases, you are I/O bound, not CPU bound,
and not memory bandwidth bound.


> IMPORTANT NOTE: Several tests here has shown that this is VERY BADLY
> affected if You have too much LAN equipment (especially VLAN seems to be
> harmful) at the edges. My speed of 960 Mbit/sec fell to 165 just by adding
> 10 feet of cable and two switches :-(

The products that Jeffrey Hsu and I and Alfred and Jon Mini
worked on at a previous company had no problems at all on a
1Gbit/S saturating the link, even through a VLAN trunk through
Cisco and one other less intelligent switch (i.e. two switches
and a VLAN trunk).

Maybe your network cards don't do hardware interrupt coelescing?
Or maybe you are sending 1 byte packets instead of MTU-sized
packets, or something?  We were using Tigon III's, with 1K packets.

-- Terry