Mysterious packets with stateful ipfw+nat

[James Halstead]

Ok, this has been driving me nuts for a while. I recently noticed that 
my 5.4-RELEASE firewall was having a problem with packet "leakage". I am 
seeing the occasional packet on the outside interface with an internal 
src ip. I put a hub between my firewall and cable modem and verified 
that the packets are indeed on the wire. Now I am in the process of 
setting up a new 6.1-RELEASE box and the same issue was happening on my 
test network.

So far I don't get it. I must be missing something obvious. At least 
everything still works in general.

The test setup is a clean install of 6.1-RELEASE, using GENERIC with the 
ipfw.ko and ipdivert.ko modules loaded. After searching around I was 
basing the configuration off of:

http://lists.freebsd.org/mailman/htdig/freebsd-ipfw/2004-June/001182.html

The test box has two Ethernet interfaces, renamed to be isp0 and net0. 
isp0 is using DHCP, and receives the address 10.42.0.220/24. net0 is 
running a DHCP server, and sits on 192.168.1.1/24. There is one single 
piece of hardware on net0 which is always assigned 192.168.1.230. The 
gateway to the actual Internet sits on 10.42.0.254. A pretty simple setup.

The internal machine is just constantly connecting to an external web 
server to generate traffic. I see the same basic type of thing happen 
for other usage as well on my main network (ssh sessions, https/http 
sessions, etc). When looking at tcpdump I am occasionally seeing (on isp0):

19:35:27.591761 aa:aa:aa:5b:db:99 > bb:bb:bb:1f:33:da,
192.168.1.230.2542 > xx.xx.53.84.80: ., cksum 0xfade (correct), 
2295591733:2295591733(0) ack 167570634 win 0

If this packet was truly supposed to be going out on the external 
interface, it should have gone through NAT and show a src ip of 
10.42.0.220. To make it more frustrating, even if I enable ifpfw at 
layer 2, I am unable to capture these rogue packets. If I watch tcpdump 
on net0 at the same time, I see the following:

19:35:27.591767 aa:aa:aa:5b:db:98 > cc:cc:cc:10:04:ce,
xx.xx.53.84.80 > 192.168.1.230.2542: ., cksum 0xfade (correct), 
913:913(0) ack 1256 win 0

The only other thing that I have noticed, is that the packets seem to 
show up on the external interface at about the same time as the dynamic 
rules expire. The dynamic rule would look like:

192.168.1.230 2542 <-> xx.xx.53.84 80

Which is pretty much what I would expect. The same setup with a 
non-stateful ipfw ruleset (using established keyword) doesn't seem to 
have this problem. Any ideas? configuration follows.


**** natd.conf ****
unregistered_only yes
dynamic yes
#deny_incoming yes
log_denied yes
log_ipfw_denied yes

(deny_incoming was set, turned it off to see if it helped but it works 
the same).

***** ipfw.rules ****
# Test stateful firewall + natd script
cmd="/sbin/ipfw add"
natout="skipto 1000"
oif="isp0"
iif="net0"
inet="192.168.1.0/24"

NOROUTE="( 172.16.0.0/12 or 192.168.0.0/16 or \
0.0.0.0/8 or 169.254.0.0/16 or 192.0.2.0/24 or 224.0.0.0/4 or 240.0.0.0/4 )"

####
# Start with a clean ruleset
/sbin/ipfw -q -f flush

####
# Allow all traffic on the loopback and internal network, to keep this 
simple.
$cmd 2 allow all from any to any via lo0
$cmd 5 allow all from any to any in via $iif
$cmd 6 allow all from any to any out xmit $iif

# Translate incoming traffic here
$cmd 200 divert natd ip from any to any in via $oif
$cmd 205 check-state

# Outbound
# Use stateful inspection to allow any connection from the internal network.
$cmd 300 $natout tcp from any to any out via $oif setup keep-state
$cmd 305 $natout udp from any to any out via $oif keep-state
$cmd 310 $natout icmp from any to any out via $oif keep-state

# Inbound
# Prevent non-routable networks on the external interface.
$cmd 400 deny all from $NOROUTE to any in via $oif

# Allow incoming DHCP for external network address assignment.
$cmd 450 allow udp from any to any 68 in via $oif keep-state

# Allow incoming SSH to this machine
$cmd 455 allow tcp from any to me 22 in via $oif setup keep-state

# Allow incoming ICMP
$cmd 460 allow icmp from any to any icmptypes 0,3,11,12 in via $oif

$cmd 999 deny log ip from any to any

# NAT rule for outgoing traffic.
$cmd 1000 divert natd ip from any to any out via $oif
$cmd 1005 allow ip from any to any

Thanks for any insight,

-James