tcp_output starving -- is due to mbuf get delay?

[Borje Josefsson]

On Thu, 10 Apr 2003 11:16:36 CDT Eric Anderson wrote:

> Mike Silbersack wrote:
> >>My hosts are connected directly to core routers in a 10Gbps nationwide
> >>network, so if anybody is interested in some testing I am more than
> >>willing to participate. If anybody produces a patch, I have a third system
> >>that I can use for piloting of that too.
> >>
> >>--Börje
> > 
> > 
> > This brings up something I've been wondering about, which you might want
> > to investigate:
> > 
> >>From tcp_output:
> > 
> > 		if (error == ENOBUFS) {
> > 	                if (!callout_active(tp->tt_rexmt) &&
> >                             !callout_active(tp->tt_persist))
> > 	                        callout_reset(tp->tt_rexmt, tp->t_rxtcur,
> >                                       tcp_timer_rexmt, tp);
> > 			tcp_quench(tp->t_inpcb, 0);
> > 			return (0);
> > 		}
> > 
> > That tcp_quench knocks the window size back to one packet, if I'm not
> > mistaken.  You might want to put a counter there and see if that's
> > happening frequently to you; if so, it might explain some loss of
> > performance.
> > 
> > Have you tried running kernel profiling yet?  It would be interesting to
> > see which functions are using up the largest amount of time.

Could do that if I knew how... Not before the weekend though, right now 
I'm at the longue at the airport...
 
> It's interesting - I'm only getting about 320mb/s.. I must be hitting a 
> similar problem.  I'm not nearly as adept at hacking code to find bugs 
> though. :(

320 Mbit/sec seems familiar, this was what I got when I first tried on a 
system with "traditional" PCI bus. Changing the OS to NetBSD on that box 
bumped that to 525 Mbit/sec. You need wide PCI (or preferrably PCI-X for 
this).

What happens in that case for me is that I run out of CPU resources. Try 
running "top" in one window and "netstat 1" in another while bashing the 
net with ttcp.

If everything is OK (which it apparently isn't), top will show free CPU, 
and netstat should show a *very* steady packet flow (around 90kpps if You 
have MTU 1500). Any packet loss is fatal for this speed, so if there is a 
way (as indicated by Mike above) to not restarting with windowsize from 
scratch that will make recovery much better.

My test was done with ttcp and this parameters:

ttcp -s -t -f m -l 61440 -n 20345 dest.host

(tuned for a 10 sec test at 1Gbps).

IMPORTANT NOTE: Several tests here has shown that this is VERY BADLY 
affected if You have too much LAN equipment (especially VLAN seems to be 
harmful) at the edges. My speed of 960 Mbit/sec fell to 165 just by adding 
10 feet of cable and two switches :-(

--Börje