stateful firewall vs. closing TCP connections

[Gordon Burditt]

I've been attempting to update my firewall rules.  The old one
worked OK with ipfw 1, but it didn't use a stateful firewall, and
ipfw 2 allows some shortcuts and has more features that I might
like to use.  I'm using FreeBSD-4.9 with IPFW2.  Yes, I've
rebuilt everything with IPFW2=TRUE

However, I've been running into a problem that the stateful firewall
seems to be closing itself too quickly.  It may not be causing
observable errors, but my logs are full of junk for denied packets
from LEGITIMATE connections.  The objective here was to look at
packets rejected under the new rules but not rejected under the old
rules before actually denying them.  Then I deny these and see only
the port scans, probes, and other bad traffic in the logs.

This set of rules actually is intended to run on both my gateway
machine and my server machine on the local LAN (with the gateway
having some extra rules to allow passthru), but I'm testing this
on the server machine only.  The server machine accepts a lot of
http, smtp, and mysql connections from outside ("outside" here is
any other machine, including my LAN), and initiates a number of
outgoing smtp (outgoing mail and Exim callout verify) and mysql
connections.  There is no NAT in this setup (at least not when I
first started working on this).  All of the machines have public
IPs.  The LAN is 100baseTX and the outside connection is DSL.

The old rules worked like this (in a section applicable to TCP
only):

20100 allow tcp from any to any established
...
21000 skipto 60000 tcp from { MYSQL_CLIENT_1, MYSQL_CLIENT_2, MYSQL_CLIENT_3} to me 3306 setup
21010 skipto 60000 tcp from me to { MYSQL_SERVER_1, MYSQL_SERVER_2} 3306 setup
...
21700 skipto 60000 tcp from me to any http setup
21710 skipto 60000 tcp from any to me http setup
...
29999 deny log all from any to any
60000 allow all from any to any


Now, this worked OK, but it's not stateful.  So, for testing purposes,
I changed it to this:

20099 check-state
20100 allow log tcp from any to any established
20110 allow log tcp from any to any not setup
...
21000 skipto 60000 tcp from { MYSQL_CLIENT_1, MYSQL_CLIENT_2, MYSQL_CLIENT_3} to me 3306 setup keep-state
21010 skipto 60000 tcp from me to { MYSQL_SERVER_1, MYSQL_SERVER_2} 3306 setup keep-state
...
21700 skipto 60000 tcp from me to any http setup keep-state
21710 skipto 60000 tcp from any to me http setup keep-state
...
29999 deny log all from any to any
60000 allow all from any to any

The rules go through a preprocessor where things like MYSQL_SERVER_*
and MYSQL_CLIENT_* get filled in with a real (and outside public)
ip.  There are a lot more rules to allow tcp setups in the full set
of rules, ending in "... setup keep-state".

Rule 20100 is not supposed to catch anything, as the established
packets should be accepted by rule 20099.  Rule 20110 only catches
packets that are neither setup nor established (and probably
malicious).  I intend to change these rules to "deny log" (or just
delete them and let rule 29999 handle it) after making sure doing
this doesn't disrupt anything.

So, what happens?  I get the usual bunch of port scans in the log.
BUT, I also get a lot of packets logged from LEGITIMATE connections
by rule 20100, which I believe are coming at the tail end of the
connection.  This happens too often for me to believe it's due to
retransmitted packets.  Also, I can match up stuff in the ipfw logs
against mail or Apache logs to see that a real connection came in.
I cannot, however, match tcpdump output against log output:  there
is not enough detail in the logs to determine which packet from that
particular server was logged.  But it seems like it's the tail end
of the connection.

I tried setting net.inet.ip.fw.dyn_rst_lifetime and
net.inet.ip.fw.dyn_fin_lifetime to 3 from 1.  No change.

Since 'log' doesn't log much of the packet details, I split rule
20100 into several rules with various combinations of tcpflags so
I could use the rule number to figure out what the flags were.  What
seems to be happening is that the SERVER side (my host acts as both
client and server at various times, and the symptoms seem to follow
the role it's playing) sends a FIN-ACK or FIN packet and the CLIENT
side sends back a RST.  If I change rules 20100-20109 to "deny log",
the server side sends two FIN-ACK packets and I don't get back the
RST, so it seems the server is retrying the packet.

I have not observed any actual problems with the transactions from
denying the packet.  But I'm not sure how many programs actually
check for errors on close.

Question:  if one side of the connection is closed, does the stateful
firewall close off both directions?  Actually, I don't see why
this should be a problem, especially with the lifetime set to 3 
seconds, since I don't think a whole http transaction even takes 3
seconds.

Is this normal behavior of a TCP connection?  Is this a firewall
bug?  Is there a way I can avoid logging these, but log the real
malicious stuff?

What's the deal with rule 60000?  At one point I was trying to deal
with two outside connections, and packets FROM public netblock A
had to go out interface de2 (using fwd) and packets FROM public
netblock B had to go out tun0.  That applies only to the gateway
anyway.

						Gordon L. Burditt