tcp_output starving -- is due to mbuf get delay?

[Sean Chittenden]

> When testing  GigE path that has 67 ms RTT, the maximum TCP throughput is
> limited at 250 Mb/s. By tracing the problem, I found that tcp_output() is
> starving
> where snd_wnd and snd_cwnd are fully open. The snd_cc is never filled beyond
> the 4.05MB even though the snd_hiwat is 10MB and snd_sbmax is 8MB. That is,
> sosend never stopped at sbwait. So only place can slow down is the mbuf
> allocation
> in sosend(). The attached trace file shows that each MGET and MCLGET takes
> significant time -- around 8 us at slow start time, and gradually increasing
> after that
> in an range 18 to 648.
> Each packet Tx on GigE takes 12 us. It average mbuf allocation takes 18 us, then
> 
> the performance will be reduced to 40%, in fact it is down to 25%, which means
> higher average delay.
> 
> I have change NMBCLUSTER from 2446 to 6566 to 10240, and nothing is improved.
> 
> Any one can tell what factors would cause MGET / MCLGET to wait?
> Is there any way to make MGET/MCLGET not to wait?

Luigi posted a patch about this a while back (last summer sometime,
iirc).

http://people.freebsd.org/~seanc/patches/#o1_mbuf_lookup

I updated his patch but haven't had a chance to test it.  If you're
feeling brave, see if applying this patch fixes this bottle neck.  -sc

-- 
Sean Chittenden