why BAD state messages

Alexandre Biancalana biancalana at gmail.com
Fri Aug 15 16:26:32 UTC 2008


On 8/15/08, Max Laier <max at love2party.net> wrote:
> On Friday 15 August 2008 16:08:38 Alexandre Biancalana wrote:
>  > Hi list,
>  >
>  >   I'm experiencing some problems with blocked connections because of
>  > bad states but I need some more information about why this is
>  > happening, if this is timeout between tcp handshake, or state creation
>  > or application trying to talk on closed connection.
>  >
>  >   I have two FreeBSD 7-STABLE with PF, carp, pfsync and max carpdev
>  > patch and two application servers (jboss) that listen on port 9090
>  > behind this firewalls, some connections from external clients off this
>  > appservers are (apparently random) being blocked, enabling loud (pfctl
>  > -x loud) I can see in /var/log/messages the following messages:
>  >
>  > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
>  > 10.10.110.34:52347 [lo=3922530250 high=3922595445 win=65535
>  > modulator=0] [lo=3059100500 high=3059158735 win=65195 modulator=0] 4:4
>  > S seq=398900533 (398900533) ack=3059100500 len=0 ackskew=0 pkts=6:20
>  > dir=in,fwd
>  > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
>  > 10.10.110.34:50668 [lo=395881033 high=395946233 win=65535 modulator=0]
>  > [lo=3568232053 high=3568290288 win=65200 modulator=0] 4:4 S
>  > seq=2480335288 (2480335288) ack=3568232053 len=0 ackskew=0 pkts=6:20
>  > dir=in,fwd
>  > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
>  > 10.10.110.34:51582 [lo=3528357041 high=3528421509 win=65535
>  > modulator=0] [lo=3809540772 high=3809605893 win=64468 modulator=0] 9:9
>  > S seq=3810516558 (3810516558) ack=3809540772 len=0 ackskew=0 pkts=6:5
>  > dir=in,fwd
>  > kernel: pf: BAD state: TCP 10.10.6.19:9090 10.10.6.19:9090
>  > 10.10.110.34:50668 [lo=395881033 high=395946233 win=65535 modulator=0]
>  > [lo=3568232053 high=3568290288 win=65200 modulator=0] 4:4 S
>  > seq=2480335288 (2480335288) ack=3568232053 len=0 ackskew=0 pkts=6:20
>  > dir=in,fwd
>  > kernel: pf: BAD state: TCP 10.10.6.18:9090 10.10.6.18:9090
>  > 10.10.81.242:2434 [lo=538716318 high=538780855 win=65535 modulator=0]
>  > [lo=1004209856 high=1004274961 win=64537 modulator=0] 4:9 S
>  > seq=1634723484 (1634723484) ack=1004209856 len=0 ackskew=0 pkts=5:4
>  > dir=in,fwd
>  >
>  > I tried to set custom tcp timeout options in this rules but this does not
>  > help
>  >
>  > pass log proto tcp from any to { $apphpr01 $apphpr02 } port { 9090 }
>  > keep state (tcp.opening 60, tcp.closed 180, tcp.finwait 90)
>  >
>  >
>  > Any ideas on how can I know why this connections are being blocked ??
>  > I can provide any additional information needed.
>
>
> The blocked packets are SYNs.  That means you are trying to reuse a port.
>  This works if the state on both sides is >= FIN_WAIT2 (9) and you have pf.c
>  r181291 (or one that has it merged).  CVS rev 1.55 or 1.46.2.3 (RELENG_7) or
>  apply the following patch:
>  http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/contrib/pf/net/pf.c.diff?r1=1.54;r2=1.55

Hi Max ! Thank you for your quick the reply.

Looking at machine's pf.c this looks too old

# grep FBSD /usr/src/sys/contrib/pf/net/pf.c
__FBSDID("$FreeBSD: src/sys/contrib/pf/net/pf.c,v 1.46.2.1 2007/11/25
19:26:46 mlaier Exp $");

I will do a csup, rebuild the kernel and see how this improve the situation.

>
>  This should fix the instances above where it says "...] 9:9 S ..."  The others
>  might be an artifact from pfsync or asymmetric routing?  You can also mitigate
>  the problem by giving your clients and the pf-forwarding a larger port range
>  for outgoing connections.  This is a typical problem if you open a large
>  number of connections from one client (or load balancer) to one server.  You
>  can only have so many open at a given time.  Check if you can enable streaming
>  mode somehow.

Looking the logs I made some math on each state

 9:9      6174 times
 4:4      3283 times
 4:9      2611 times
10:10   1382 times
 2:0        878 times
 9:4        520 times

How can I give a larger range for outgoing conections if the clients
connect directly to the servers ? In this case I don't have any rdr
rule.

Thank you again!


More information about the freebsd-pf mailing list