Should syncache.count ever be negative?

[Mike Silbersack]

On Sat, 10 Nov 2007, Mike Silbersack wrote:

> FWIW, my crazy theory of the moment is this:  We have some bug that happens 
> when the listen queues overflow in 7.0, and your test is strenuous enough to 
> hit the listen queue overflow condition, leading to total collapse.  I'll 
> have to cobble together a test program to see what happens in the listen 
> queue overflow case.

Post testing, I have a different theory.

Can you also try

sysctl net.inet.tcp.syncookies=0

I modified netrate's httpd to sleep a lot and found an interesting 
behavior between listen queue overflows and syncookies:

04:28:21.470931 IP 10.1.1.8.50566 > 10.1.1.6.http: S 287310302:287310302(0) win 32768 <mss 1460,nop,wscale 0,nop,nop,timestamp 3293131923 0,sackOK,eol>
04:28:21.470939 IP 10.1.1.6.http > 10.1.1.8.50566: S 4209413098:4209413098(0) ack 287310303 win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 1264205330 3293131923>
04:28:21.473487 IP 10.1.1.8.50566 > 10.1.1.6.http: . ack 1 win 33304 <nop,nop,timestamp 3293131926 1264205330>
04:28:21.473493 IP 10.1.1.6.http > 10.1.1.8.50566: R 4209413099:4209413099(0) win 0
04:28:21.473642 IP 10.1.1.8.50566 > 10.1.1.6.http: P 1:78(77) ack 1 win 33304 <nop,nop,timestamp 3293131926 1264205330>
04:28:21.482555 IP 10.1.1.6.http > 10.1.1.8.50566: P 1:126(125) ack 78 win 8326 <nop,nop,timestamp 1264205339 3293131926>
04:28:21.482563 IP 10.1.1.6.http > 10.1.1.8.50566: F 126:126(0) ack 78 win 8326 <nop,nop,timestamp 1264205339 3293131926>
04:28:21.487047 IP 10.1.1.8.50566 > 10.1.1.6.http: R 287310380:287310380(0) win 0
04:28:21.487398 IP 10.1.1.8.50566 > 10.1.1.6.http: R 287310380:287310380(0) win 0

The listen queue overflow causes the socket to be closed and a RST sent, 
but the next packet from 10.1.1.8 crosses it on the wire and activates the 
syncookie code, reopening the connection.  Meanwhile, the RST arrives at 
10.1.1.8 and closes its socket, leading to it sending RSTs when the data 
from 10.1.1.6 arrives.

Not sure if that's your problem or not, but it's interesting.

-Mike