tcp_output starving -- is due to mbuf get delay?

[Hiten Pandya]

[ cross post to hackers removed, thanks ]

Mike Silbersack (Thu, Apr 10, 2003 at 11:44:42AM -0500) wrote:
> 
> On Thu, 10 Apr 2003, Borje Josefsson wrote:
> 
> > What we did in NetBSD (-current) was to increase IFQ_MAXLEN in (their)
> > sys/net/if.h, apart from that it's only "traditional" TCP tuning.
> >
> > My hosts are connected directly to core routers in a 10Gbps nationwide
> > network, so if anybody is interested in some testing I am more than
> > willing to participate. If anybody produces a patch, I have a third system
> > that I can use for piloting of that too.
> >
> > --B?rje
> 
> This brings up something I've been wondering about, which you might want
> to investigate:
> 
> >From tcp_output:
> 
> 		if (error == ENOBUFS) {
> 	                if (!callout_active(tp->tt_rexmt) &&
>                             !callout_active(tp->tt_persist))
> 	                        callout_reset(tp->tt_rexmt, tp->t_rxtcur,
>                                       tcp_timer_rexmt, tp);
> 			tcp_quench(tp->t_inpcb, 0);
> 			return (0);
> 		}
> 
> That tcp_quench knocks the window size back to one packet, if I'm not
> mistaken.  You might want to put a counter there and see if that's
> happening frequently to you; if so, it might explain some loss of
> performance.

Maybe something like this:

%%%
Index: sys/netinet/tcp_output.c
===================================================================
RCS file: /home/ncvs/src/sys/netinet/tcp_output.c,v
retrieving revision 1.78
diff -u -r1.78 tcp_output.c
--- sys/netinet/tcp_output.c	19 Feb 2003 22:18:05 -0000	1.78
+++ sys/netinet/tcp_output.c	12 Apr 2003 19:52:31 -0000
@@ -930,6 +930,7 @@
                             !callout_active(tp->tt_persist))
 	                        callout_reset(tp->tt_rexmt, tp->t_rxtcur,
                                       tcp_timer_rexmt, tp);
+			tcpstat.tcps_selfquench++;
 			tcp_quench(tp->t_inpcb, 0);
 			return (0);
 		}
Index: sys/netinet/tcp_var.h
===================================================================
RCS file: /home/ncvs/src/sys/netinet/tcp_var.h,v
retrieving revision 1.88
diff -u -r1.88 tcp_var.h
--- sys/netinet/tcp_var.h	1 Apr 2003 21:16:46 -0000	1.88
+++ sys/netinet/tcp_var.h	12 Apr 2003 19:52:31 -0000
@@ -394,6 +394,8 @@
 	u_long	tcps_sc_zonefail;	/* zalloc() failed */
 	u_long	tcps_sc_sendcookie;	/* SYN cookie sent */
 	u_long	tcps_sc_recvcookie;	/* SYN cookie received */
+
+	u_long	tcps_selfquench;	/* self-quench count */
 };
 
 /*
Index: usr.bin/netstat/inet.c
===================================================================
RCS file: /home/ncvs/src/usr.bin/netstat/inet.c,v
retrieving revision 1.58
diff -u -r1.58 inet.c
--- usr.bin/netstat/inet.c	2 Apr 2003 20:14:44 -0000	1.58
+++ usr.bin/netstat/inet.c	12 Apr 2003 19:52:32 -0000
@@ -389,6 +389,7 @@
 	p(tcps_sndprobe, "\t\t%lu window probe packet%s\n");
 	p(tcps_sndwinup, "\t\t%lu window update packet%s\n");
 	p(tcps_sndctrl, "\t\t%lu control packet%s\n");
+	p(tcps_selfquench, "\t\t%lu send%s resulting in self-quench\n");
 	p(tcps_rcvtotal, "\t%lu packet%s received\n");
 	p2(tcps_rcvackpack, tcps_rcvackbyte, "\t\t%lu ack%s (for %lu byte%s)\n");
 	p(tcps_rcvdupack, "\t\t%lu duplicate ack%s\n");
%%%

Cheers.

  -- Hiten