why BAD state messages

[Max Laier]

On Friday 15 August 2008 16:08:38 Alexandre Biancalana wrote:
> Hi list,
>
>   I'm experiencing some problems with blocked connections because of
> bad states but I need some more information about why this is
> happening, if this is timeout between tcp handshake, or state creation
> or application trying to talk on closed connection.
>
>   I have two FreeBSD 7-STABLE with PF, carp, pfsync and max carpdev
> patch and two application servers (jboss) that listen on port 9090
> behind this firewalls, some connections from external clients off this
> appservers are (apparently random) being blocked, enabling loud (pfctl
> -x loud) I can see in /var/log/messages the following messages:
>
> kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
> 10.10.110.34:52347 [lo=3922530250 high=3922595445 win=65535
> modulator=0] [lo=3059100500 high=3059158735 win=65195 modulator=0] 4:4
> S seq=398900533 (398900533) ack=3059100500 len=0 ackskew=0 pkts=6:20
> dir=in,fwd
> kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
> 10.10.110.34:50668 [lo=395881033 high=395946233 win=65535 modulator=0]
> [lo=3568232053 high=3568290288 win=65200 modulator=0] 4:4 S
> seq=2480335288 (2480335288) ack=3568232053 len=0 ackskew=0 pkts=6:20
> dir=in,fwd
> kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
> 10.10.110.34:51582 [lo=3528357041 high=3528421509 win=65535
> modulator=0] [lo=3809540772 high=3809605893 win=64468 modulator=0] 9:9
> S seq=3810516558 (3810516558) ack=3809540772 len=0 ackskew=0 pkts=6:5
> dir=in,fwd
> kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
> 10.10.110.34:50668 [lo=395881033 high=395946233 win=65535 modulator=0]
> [lo=3568232053 high=3568290288 win=65200 modulator=0] 4:4 S
> seq=2480335288 (2480335288) ack=3568232053 len=0 ackskew=0 pkts=6:20
> dir=in,fwd
> kernel: pf: BAD state: TCP 10.10.6.18:9090 10.10.6.18:9090
> 10.10.81.242:2434 [lo=538716318 high=538780855 win=65535 modulator=0]
> [lo=1004209856 high=1004274961 win=64537 modulator=0] 4:9 S
> seq=1634723484 (1634723484) ack=1004209856 len=0 ackskew=0 pkts=5:4
> dir=in,fwd
>
> I tried to set custom tcp timeout options in this rules but this does not
> help
>
> pass log proto tcp from any to { $apphpr01 $apphpr02 } port { 9090 }
> keep state (tcp.opening 60, tcp.closed 180, tcp.finwait 90)
>
>
> Any ideas on how can I know why this connections are being blocked ??
> I can provide any additional information needed.

The blocked packets are SYNs.  That means you are trying to reuse a port.  
This works if the state on both sides is >= FIN_WAIT2 (9) and you have pf.c 
r181291 (or one that has it merged).  CVS rev 1.55 or 1.46.2.3 (RELENG_7) or 
apply the following patch: 
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/contrib/pf/net/pf.c.diff?r1=1.54;r2=1.55

This should fix the instances above where it says "...] 9:9 S ..."  The others 
might be an artifact from pfsync or asymmetric routing?  You can also mitigate 
the problem by giving your clients and the pf-forwarding a larger port range 
for outgoing connections.  This is a typical problem if you open a large 
number of connections from one client (or load balancer) to one server.  You 
can only have so many open at a given time.  Check if you can enable streaming 
mode somehow.

-- 
/"\  Best regards,                      | mlaier at freebsd.org
\ /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.love2party.net/  | mlaier at EFnet
/ \  ASCII Ribbon Campaign              | Against HTML Mail and News