PF/FreeBSD 6 and FIN_WAIT2 TCP exhaustion

[Olivier Warin]

Le 2 janv. 06 à 18:30, Daniel Hartmeier a écrit :

> On Mon, Jan 02, 2006 at 05:18:30PM +0100, TYBERGHIEN Eric TRANSPAC  
> wrote:
>
>> Can you help me to solve this feature. Is it a bug, a mechanism of  
>> DOS
>> auto-protection or a mis-understood of the PF features ?
>
> Look at the TCP RFC, sections "Knowing When to Keep Quiet" and "The  
> TCP
> Quiet Time Concept" starting on page 27 of
>
>   http://www.faqs.org/rfcs/rfc793.html
>
> After a client closes the connection to a server, it may not re-use  
> the
> same source port (to the same server port) before a quiet period has
> passed. This is designed so packets from the first connection arriving
> late at the server (due to taking different routes) can't disturb the
> second connection.
>
> This obviously limits the (sustained) rate at which your client can
> connect to the server (to 65536 connections per 90 seconds, no matter
> how fast your network may be). This was probably considered more than
> enough when the TCP RFC was written, but nowadays people expect higher
> connection rates in this case.

RFC793 was written in the 80's ;-)

> The reasonable thing would be send
> multiple HTTP requests over one persistent connection, since the
> overhead of establishing (and tearing down) all those connections,  
> for a
> single request each, is significant. But yours is a benchmark, and  
> not a
> real application protocol, so I guess that's beside the point. :)
>
> FreeBSD (and other OS) re-use ports from connections in TIME_WAIT  
> state
> when they need to. The assumption is that the disadvantage of not
> detecting late arrivals of earlier connections is outweighed by the
> increased connection rate possible.
>
> You can tell pf to purge states in TIME_WAIT earlier, too. Those  
> 90s are
> merely the default,
>
>   $ pfctl -st
>   tcp.closed                   90s

Then he may reach another limits, kernel default values, not pf  
states ones.
net.inet.tcp.msl (10000ms by default) can be lower for production use.
RFC793, page 23 says sockets stay in a TIME_WAIT state two times the  
msl value.

Page 28:
"  To be sure that a TCP does not create a segment that carries a
   sequence number which may be duplicated by an old segment  
remaining in
   the network, the TCP must keep quiet for a maximum segment lifetime
   (MSL) before assigning any sequence numbers upon starting up or
   recovering from a crash in which memory of sequence numbers in use  
was
   lost.  For this specification the MSL is taken to be 2 minutes.  This
   is an engineering choice, and may be changed if experience indicates
   it is desirable to do so. "
[...]
"   TCP segments for at least the agreed Maximum Segment Lifetime (MSL)
     in the internet system of which the host is a part.  In the
     paragraphs below, an explanation for this specification is given.
     TCP implementors may violate the "quiet time" restriction, but only
     at the risk of causing some old data to be accepted as new or new
     data rejected as old duplicated by some receivers in the internet
     system."

Nevertheless, please consider that this can prohibe high latency  
connections to reach your servers, especially if your customers are  
transiting by operators with bad peerings agreement.
On the other hand, MSL was, at the origin, arbitrary defined...

> and you can change it, either globally with 'set timeout tcp.closed  
> 15'
> or per rule with 'keep state (tcp.closed 15)'. Note that purging only
> occurs in intervals (default 10s), so if you set the timeout to 15s, a
> state may be purged after 15+10s. If you need higher resolution, lower
> the interval (set timeout interval 5).
--
Olivier Warin - http://xview.net
Stay connected !