State Mismatch and tcp.closed

[Martin Turgeon]

Max Laier a écrit :
> Can you please post your ruleset.  I suspect there is something wrong with
> it.  By the way, I noticed that your are using a 127/8 addresse for your
> web server.  Are you - by chance - running in a jail of kinds?  In that
> case you might need "set skip on lo0" to avoid troubles.  Depending on the
> kind of filtering you are doing this might be complicated, however.
> 
> In any case, we'd need more details about your setup to help.
> 

I'm not too inclined to post my ruleset to the whole list, so I propose 
to send it only to people interested in it, in a private mail.

Thanks a lot Max for your interest in my problem. I'm sending my pf.conf 
to you in a few moments.

Yes, I'm using jails to isolate every services and they are all binded 
on the loopback interface. I'm using RDR and NAT to make them available 
from the outside.

Martin

> Am Fr, 29.05.2009, 17:32, schrieb Martin Turgeon:
>> Martin Turgeon a écrit :
>>> Hi list!
>>>
>>> I had a problem with state mismatch on my DB server that I solved by
>>> lowering the tcp.closed timeout. I setted it to 2 instead of 90.
>>>
>>> I now have what looks like the same problem on the front-end web server.
>>> However, when I tried to apply the same fix, I got connection problem
>>> with the back-end DB, but the state mismatch disappearred.
>>>
>>> On the front-end web server, the state mismatch occurs on the external
>>> interface, only on port 80.
>>>
>>> I enabled misc debugging and got this in /var/log/messages on the
>>> front-end web server:
>>>
>>> May 28 05:02:19 francis kernel: pf: BAD state: TCP 127.0.0.25:80
>>> 206.125.166.65:80 98.207.239.10:54737 [lo=820536733 high=820603340
>>> win=65535 modulator=0 wscale=0] [lo=2871317100 high=2871375106 win=8326
>>> modulator=0 wscale=3] 7:4 R seq=820536733 (820536732) ack=2871317100
>>> len=0 ackskew=0 pkts=43:69 dir=in,fwd
>>> May 28 05:02:19 francis kernel: pf: State failure on:         |
>>> May 28 05:02:19 francis kernel: pf: BAD state: TCP 127.0.0.25:80
>>> 206.125.166.65:80 98.207.239.10:54733 [lo=374985971 high=375052578
>>> win=65535 modulator=0 wscale=0] [lo=2999164748 high=2999229169 win=8326
>>> modulator=0 wscale=3] 7:4 R seq=374985971 (374985970) ack=2999164748
>>> len=0 ackskew=0 pkts=40:54 dir=in,fwd
>>> May 28 05:02:19 francis kernel: pf: State failure on:         |
>>> May 28 05:03:06 francis kernel: pf: BAD state: TCP 127.0.0.20:80
>>> 206.125.166.80:80 123.116.84.41:59776 [lo=3407758259 high=3407823796
>>> win=4096 modulator=0 wscale=2] [lo=374200006 high=374216390 win=8192
>>> modulator=0 wscale=3] 4:2 A seq=3407758259 (3407758260) ack=2320196160
>>> len=0 ackskew=-1945996154 pkts=1:1 dir=in,fwd
>>> May 28 05:03:06 francis kernel: pf: State failure on:     3   |
>>> May 28 05:03:06 francis kernel: pf: BAD state: TCP 127.0.0.20:80
>>> 206.125.166.80:80 123.116.84.41:59776 [lo=3407758259 high=3407823796
>>> win=4096 modulator=0 wscale=2] [lo=374200006 high=374216390 win=8192
>>> modulator=0 wscale=3] 4:2 RA seq=3407758259 (3407758260) ack=2320196160
>>> len=0 ackskew=-1945996154 pkts=1:1 dir=in,fwd
>>>
>>> This server has been up for 12 days and already got almost 600000 state
>>> mismatch!
>>>
>>> I tried to lower tcp.finwait, no result. I tried to set optimization to
>>> aggressive, no result. I tried to disable port randomization via sysctl,
>>> no result either.
>>>
>>> I tcpdumped and there is only a few RST so I don't understand why
>>> tcp.closed would solve my problem. If it's a problem with source port
>>> reuse, tcp.finwait should be the timeout that would help, not
>>> tcp.closed, right?
>>>
>>> How can a lower tcp.closed on the front-end cause mysql connection
>>> problem with the back-end? I tcpdumped while there is a connection
>>> problem with the DB and there is nothing that seems wrong, no RST at
>>> all! The front-end web server tries to connect to the DB, wait 3 sec and
>>> if it fails to establish a connection, it then tries to connect to a
>>> read-only backup DB, on another server, which never fails to connect.
>>>
>>> The only thing I'm sure is that it's the tcp.closed that cause the DB
>>> connection problem. As soon as I remove it, the state mismatch comes
>>> back on the external interface but there's no DB connection problem
>>> anymore.
>>>
>>> What am I missing?
>>>
>>> Martin
>>>
>> I forgot to mention in the starting post what version I'm using:
>>
>> uname -a on the front-end web server:
>> FreeBSD webserver 7.2-RELEASE FreeBSD 7.2-RELEASE #0: Fri May  1
>> 07:18:07 UTC 2009
>> root at driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC  amd64
>>
>> uname -a on the back-end MySQL server:
>> FreeBSD mysql 7.0-RELEASE-p5 FreeBSD 7.0-RELEASE-p5 #1: Tue Oct  7
>> 09:57:31 EDT 2008 root at martin.ringadmin.com:/usr/obj/usr/src/sys/OPTIK
>> amd64
>>
>> I read about the port reuse problem when I first experienced it with the
>> DB server and I saw that this wasn't going to happen with the new
>> release. I were happy to build I new 7.2-Rel server so that I wasn't
>> going to face the same problem.
>>
>> But, in fact, I'm facing what looks like the same problem...
>>
>> I'm all ears to any pointers/suggestions!
>>
>> Thanks for your precious help.
>>
>> Martin
>> _______________________________________________
>> freebsd-pf at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-pf
>> To unsubscribe, send any mail to "freebsd-pf-unsubscribe at freebsd.org"
>>
>>
>> !DSPAM:4a200026570535209328925!
>>
>>
> 
>