kern/86871: allocation logic for PCBs in TIME_WAIT state causes packet drops on stateful FWs

[Vladimir Kotal]

>Number:         86871
>Category:       kern
>Synopsis:       allocation logic for PCBs in TIME_WAIT state causes packet drops on stateful FWs
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Oct 03 12:10:10 GMT 2005
>Closed-Date:
>Last-Modified:
>Originator:     Vladimir Kotal
>Release:        5.4-RELEASE-p7
>Organization:
>Environment:
FreeBSD ipfw2 5.4-RELEASE-p7 FreeBSD 5.4-RELEASE-p7 #7: Mon Oct  3 11:48:16 CEST 2005     lada at ipfw2:/sys/i386/compile/GENERIC  i386
>Description:
      From FreeBSD 5.x on, it is possible to allocate local ports for TCP connections even if PCB for given socket is in TIME_WAIT state. This causes execessive packet drops in scenarios where client initiates lots of outgoing connections toward server with stateful firewall (pf).

This behavior is not present in 4.x releases.
>How-To-Repeat:
1. download and compile
   http://techie.devnull.cz/public/tcpclient-bind.c
   [ or use any other basic TCP client which can bind(2) local port ]
2. connect to TCP service running on server with stateful firewall.
3. wait for tcpclient-bind to end
4. display socket state for the connection from step nr. 2
5. connect to the same TCP service
6. watch server logs (dmesg) for 'pf BAD state' entries

For example:

lada at ipfw2$ ./tcpclient-bind SERVER 21
connecting
connected
src addr = CLIENT-IP
src port = 1234
closing connection
lada at ipfw2$ netstat -an |grep SERVER-IP
tcp4       0      0  CLIENT-IP.1234     SERVER-IP.21      TIME_WAIT
lada at ipfw2$ ./tcpclient-bind SERVER 21
connecting
tcpclient-bind: connect: Operation timed out
lada at ipfw2$ 

the 'BAD state' messages then look like this:

Oct  3 13:53:54 SERVER kernel: pf: BAD state: TCP SERVER-IP:21 SERVER-IP:21 CLIENT-IP:4094 [lo=1074578133 high=1074644741 win=33304 modulator=0 wscale=1] [lo=819316103 high=819382710 win=33304 modulator=0 wscale=1] 9:9 S seq=1087912989 ack=819316103 len=0 ackskew=0 pkts=12:19 dir=in,fwd
Oct  3 13:53:54 SERVER kernel: pf: State failure on: 1       | 5  


example of stateful rule:

  pass in quick on $ext_if proto tcp from any to ($ext_if) \
        port = ftp \
        flags S/S keep state \
        label "EXT-ingress--ftp-cmd"


This problem is always reproducible.

One of the scenarios where this problems could manifest itself is e.g. proxy+backend setup where proxy issues big number of connections to backend server running stateful firewall.
>Fix:

There is number of possibilities:

- convert stateful firewall ruleset to stateless (really not an option in my setup)
- tweak timeouts for firewall states (also not an option)
- remove INP_TIMEWAIT logic from FreeBSD 5.x TCP stack altogether
- use following patch:
  http://techie.devnull.cz/public/noTW.patch
  - this patch adds new sysctl net.inet.ip.portrange.twalloc which can be used to revert the INP_TIMEWAIT behavior to the old state. the patch will preserve current bind(2)/connect(2) behavior in FreeBSD 5.x by default.



>Release-Note:
>Audit-Trail:
>Unformatted: