Large number of http connections immediately dropped

[Robert Watson]

On Wed, 30 Jul 2008, Alexander Strange wrote:

> On Jul 21, 2008, at 3:53 PM, Ivan Voras wrote:
>
>> Alexander Strange wrote:
>> 
>>> And there's no firewalls or packet shapers in front of it.
>> 
>> How about on it? Do you run ipfw?
>
> No, I wouldn't answer a question so specifically like that.
>
> We didn't see this problem after recompiling without SMP support and waiting 
> for a day or two, but that immediately brought the load average up to around 
> 50 and made it much slower, so that's clearly not a solution. It also really 
> doesn't make me look forward to debugging it...
>
> (Disabling net.isr.direct and some other things didn't seem to have any 
> effect)

Turning off SMP is probably slowing the transaction rate down sufficiently 
that you're not seeing the problem.  The reason to ask the firewall question 
(ipfw, pf, etc) is that as the rate of TCP connections goes up, and if there 
are a small number of addresses involved, the reuse rate for TCP/IP 
port/address tuples becomes very high, which can cause connections to reuse 
tuples too quickly.  Sometimes firewalls are more sensitive to this than the 
stack -- especially if those firewalls are doing things like randomizing port 
numbers, TCP sequence numbers, etc, so in the past there have been reports 
(and bug fixes) along those lines.  I may have missed you answering this 
already, but are there a large number of remote endpoints (unique IP 
addresses) or a small one?  Such problems have come up in the past especially 
when there is a load balancer or proxy in front, as that reduces what starts 
out as a large number of hosts to a very small number (exactly one).

Robert N M Watson
Computer Laboratory
University of Cambridge