NAT/ipfw blocking internal traffic

[Ian Smith]

On Thu, 31 Oct 2013 13:10:42 -0700, Casey Scott wrote:
 > Hello,
 > 
 > My NAT and ipfw ruleset follow almost exactly what is given at
 > http://www.freebsd.org/doc/handbook/firewalls-ipfw.html

Almost, but perhaps not quite near enough.  Firstly, I'd normally advise 
largely ignoring the handbook section on ipfw; it's wrong-headed in more 
ways than I care to detail, and contains a number of factual errors 
that if taken as true will impede learning about ipfw.  I'd instead 
advise using the appropriate ruleset from /etc/rc.firewall, in this case 
apparently the 'client' ruleset, however let's have a look at it ..

 > The problem I'm encountering is that a portion of my outbound internal
 > traffic is being blocked by ipfw. This is a fresh Freebsd installaion, so
 > I'm kind of at a loss since the config matches the handbook. Any suggestions
 > are appreciated.
 > 
 > uname -a
 > ***********************************************
 > FreeBSD hostname 9.2-RELEASE FreeBSD 9.2-RELEASE #6 r256447: Fri Oct 18
 > 20:06:53 PDT 2013 root at hostname:/usr/src/sys/amd64/compile/hostname
 > amd64
 > ***********************************************
 > 
 > /var/log/security:
 > ***********************************************
 > Oct 29 10:14:46 hostname kernel: ipfw: 450 Deny TCP 65.126.84.81:80
 > 192.168.1.6:61681 in via fxp0
 > Oct 29 10:14:47 hostname kernel: ipfw: 450 Deny TCP 65.126.84.81:80
 > 192.168.1.6:61681 in via fxp0
 > Oct 29 10:14:47 hostname kernel: ipfw: 450 Deny TCP 65.126.84.81:80
 > 192.168.1.6:61681 in via fxp0
 > Oct 29 10:14:54 hostname kernel: ipfw: 450 Deny TCP 192.168.1.6:61915
 > 174.129.210.177:80 out via fxp0
 > Oct 29 10:17:55 hostname kernel: ipfw: 450 Deny TCP 192.168.1.6:61876
 > 65.126.84.88:80 out via fxp0
 > Oct 29 10:17:55 hostname kernel: ipfw: 450 Deny TCP 192.168.1.6:61877
 > 65.126.84.88:80 out via fxp0
 > Oct 29 10:17:58 hostname kernel: ipfw: 450 Deny TCP 192.168.1.6:61921
 > 208.85.40.45:80 out via fxp0
 > Oct 29 10:17:58 hostname kernel: ipfw: 450 Deny TCP 192.168.1.6:61921
 > 208.85.40.45:80 out via fxp0
 > ***********************************************

Ok, first to confirm that 192.168.1.6 the address of your box, and not 
another box on your LAN?  ie, you are not doing NAT as gateway for your 
LAN, but only for your single box?  No gateway_enable set in rc.conf?

How are you connecting to your upstream gateway / ISP?

 > firewall script:
 > ***********************************************
 > #!/bin/sh
 > cmd="ipfw -q add"
 > skip="skipto 500"
 > pif=fxp0
 > ks="keep-state"
 > good_tcpo="22,25,37,43,53,80,443"
 > 
 > ipfw -q -f flush
 > 
 > $cmd 002 allow all from any to any via em0 # exclude LAN traffic

That would be no good if you were acting as the gateway for your LAN.

 > $cmd 003 allow all from any to any via lo0 # exclude loopback traffic
 > 
 > $cmd 100 divert natd ip from any to any in via $pif
 > $cmd 101 check-state
 > 
 > # Authorized outbound packets
 > $cmd 136 $skip udp from any to any 53 out via $pif $ks
 > $cmd 150 $skip tcp from any to any $good_tcpo out via $pif setup $ks
 > $cmd 151 $skip icmp from any to any out via $pif $ks
 > $cmd 152 $skip udp from any to any 123 out via $pif $ks
 > 
 > # Deny all inbound traffic from non-routable reserved address spaces
 > $cmd 300 deny all from 192.168.0.0/16 to any in via $pif #RFC 1918 private
 > IP
 > $cmd 301 deny all from 172.16.0.0/12 to any in via $pif #RFC 1918 private
 > IP
 > $cmd 302 deny all from 10.0.0.0/8 to any in via $pif #RFC 1918 private
 > IP
 > $cmd 303 deny all from 127.0.0.0/8 to any in via $pif #loopback
 > $cmd 304 deny all from 0.0.0.0/8 to any in via $pif #loopback
 > $cmd 305 deny all from 169.254.0.0/16 to any in via $pif #DHCP auto-config
 > $cmd 306 deny all from 192.0.2.0/24 to any in via $pif #reserved for docs
 > $cmd 307 deny all from 204.152.64.0/23 to any in via $pif #Sun cluster
 > $cmd 308 deny all from 224.0.0.0/3 to any in via $pif #Class D & E
 > multicast

Ok, but see below re missing rule ..

 > # Authorized inbound packets
 > $cmd 400 allow tcp from any to me 76 in via $pif setup limit src-addr 2
 > $cmd 402 allow ip from any to me 53 in via $pif setup limit src-addr 2

If you're running DNS for external hosts, you need to allow both tcp 53 
and udp 53 in.  'ip' (or 'all') doesn't specify ports, and 'setup' 
only applies to tcp, so that's wrong.  If not serving DNS, drop this.

 > $cmd 420 allow tcp from any to me 80 in via $pif setup limit src-addr 2
 > $cmd 421 allow tcp from any to me 80 in via $pif setup limit src-addr 2

These are duplicates.  And 'limit src-addr 2' is silly for web traffic; 
any webpage you serve with say text and several images will likely want 
at least several ccncurrent connections, many people run 'download 
helpers' that open one connection per item.  Change '2' to say '10', at 
least while testing.  Even 50 connections hardly constitutes a DoS.

Similarly, Joe's recommended logging limit of 5 is silly.  Try 50 or the 
default 100, again at least while testing.  Add 'log' to any rules you 
want to see working (pass or deny), and use the logging to watch and 
learn about the various flows, until satisfied all is working properly.

Part of what's wrong here may be that you've left out an important rule 
to use with an all-stateful ruleset (which I personally think overkill, 
but sticking with this method ..) is one that you've just missed; I'll 
quote it from the block that comes BEFORE '# Authorised inbound pkts':

# Deny any late arriving packets # DON'T DO IT, BAD ADVICE esp. UDP DNS:
### $cmd 330 deny all from any to any frag in via $pif

# Deny ACK packets that did not match the dynamic rule table 
$cmd 332 deny tcp from any to any established in via $pif 

Some of those packets in your log may be of this type, and again may 
come from the 'limit 2' .. all subsequent connections will be refused.  
However, that doesn't explain why some of your outbound connections are 
being denied, from above ..
=======
 > Oct 29 10:14:54 hostname kernel: ipfw: 450 Deny TCP 192.168.1.6:61915
 > 174.129.210.177:80 out via fxp0
 > Oct 29 10:17:55 hostname kernel: ipfw: 450 Deny TCP 192.168.1.6:61876
 > 65.126.84.88:80 out via fxp0
=======

.. which can only get here by falling through ..

 > $cmd 450 deny log ip from any to any

.. which isn't right, by that ruleset, it should be:

# Reject & Log all unauthorized out going connections to the public Internet
$cmd 450 deny log all from any to any out via $pif 

Big difference .. your 450 is catching ALL traffic, in and out anywhere, 
not just 'out via $pif', which really should be 'out xmit $pif' anyway.

And yet, I'm still bemused as to why that traffic wasn't passed to rule 
500 by rule 150, so you might try fixing those things, adding more 
logging (eg on rule 150), and trying again; report more logging on a 
couple of controlled tests so it's clear what's happening.
 
 > # This is skipto location for outbound stateful rules
 > $cmd 500 divert natd ip from any to any out via $pif

which should be followed by, from that ruleset:

$cmd 801 allow ip from any to any	# IMPORTANT if deny by default

# Everything else is denied by default 
# deny and log all packets that fell through to see what they are
$cmd 999 deny log all from any to any	# ADVISABLE

 > ***********************************************
 > 
 > natd run options:
 > ***********************************************
 > /sbin/natd -dynamic -m -n fxp0
 > ***********************************************

Sadly that section is also well out of date; modern advice is to use 
in-kernel NAT rather than natd(8), but you'd need to refer to ipfw(8) 
for that.  NATD still works, so that's unlikely an issue here.

HTH, Ian