Kernel NAT issues

[Ian Smith]

On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote:
 > Hi Ian,
 > 
 > Thank you for your response.
 > 
 > I didnÿÿt post my ruleset because I should be able to fix the issue 
 > myself but I see now that my request to explain ÿÿhow NAT worksÿÿ was 
 > incorrect.
 > 
 > I have now included my ruleset below (as well as my initial email).

Hi Nathan,

I was really hoping someone who knows more about stateful rule handling 
(and jail networking) might have a go at this.  Oh well I'll try, but 
I'm a lousy mindreader, and really don't know which of the below 
constitutes 'hairpin NAT'.  Perhaps showing your 'netstat -finet -an' 
and 'netstat -finet -rn' may shed light on routing?  And 'ifconfig'?

 > # Enable NAT
 > ipfw nat 1 config ip $jip same_ports log

I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that 
WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed 
to you, and so traffic to it won't be subject to NAT .. correct?  But 
the WWW... address and all 10.0/16 addresses are jails, not any separate 
boxes you gateway for, right?  Just the one external interface, right?

 > 00005 allow ip from any to any via lo0
 > 00006 deny ip from any to not me in via bce0
 > 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
 > 00101 check-state

Ok, inbound from WAN is nat'd and existing stateful flows followed by 
executing the rule that originally kept state.  Where this is a skipto, 
skipto will be performed.  But where it's a nat rule, I've no idea .. 
see below, but you really don't want to add keep-state (again) there.

 > 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ recv bce0 keep-state

Hmm.  I'd limit this to perhaps icmptypes 0,3,8,11 - though a stateless 
rule would make more sense especially for inbound ICMP.  But moving on ..

 > 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ dst-port 65222 recv bce0 setup keep-state

Ok, but showting why plain text works better than HTML on lists :)

 > 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any xmit bce0 keep-state
 > 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,80,443,22,65222 xmit bce0 setup keep-state
 > 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,123 xmit bce0 keep-state

Smells ok.

 > 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
 > 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state

Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's 
got a problem .. see below.  These two are inbound traffic (recv) and as 
is, skipping to 65501 will fall through two outbound rules to be denied.

Either allow them here directly, or likely better, skipto a separate
target that then allows (or denies) them, if that's what you intended?

 > 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 setup keep-state
 > 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state

Ok, this traffic does needs to be NAT'd on the way out.

 > 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup keep-state
 > 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup keep-state
 > 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state

Not clear why these tcp ports are open inbound and outbound?  Presumably 
this is jail-to-jail traffic?  Perhaps not relevant to your problem.

 > 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup keep-state
 > 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup keep-state
 > 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
 > 65500 deny log ip from any to any

Ok.

 > 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state

This the target for outbound traffix, xmit bce0, so nat is appropriate.  
Does jail-to-jail traffic travels via lo1?  Or what?

This won't do anything to inbound traffic, but that really shouldn't get 
here except returns as the result of check-state - not from 120 & 121.

But keep-state is not ok, state was already set on the skipto.  I don't 
know how this extra keep-state might behave - does anyone have an idea?

Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic.

 > 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state

So, only remaining traffic is outbound from the host itself, and traffic 
that is to 10.0/16, but not from AAA... is to be dropped, correct?

I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp 
states .. myself, I'd go for separate rules for each eg tcp, udp, .. and 
I'd do it somewhere else than as a fall through from outbound nat rule, 
it's confusing here, to me anyway .. unless I've missed the reason?

 > 65534 deny log ip from any to any
 > 65535 deny ip from any to any


Ok, now for your demo of the problem from the later mail, which I've 
reformated to quote properly, so:

 > To further illustrate my issue, this is a small log output.
 >
 > I am running host google.com <http://google.com/> in the jail, which 
 > has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state 
 > rule.

I see you don't have logging on 101 above now.  Probably best.

 > I would expect the first piece of traffic out would be UNKNOWN 
 > (does not have an entry in the state table) but it seems the 
 > returning traffic is also showing as UNKNOWN (the second 101).

I've never logged a check-state, but UNKNOWN may not mean that ..

 > You can see that the traffic is returning on the same port it went 
 > out on, so its obviously the returning traffic. I am not sure why 
 > state is not being kept?

Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, so 
it's been correctly NAT'd on the way in.  Get rid of that keep-state on 
the nat rule at 65501 and see if not creating double entries in the 
state table helps.  And change the skipto target on 120 & 121 to only 
pass outbound traffic to outbound NAT rule/s.

Once you've done outbound NAT, probably best just to 'allow [log] all'?

 > Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
 > Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
 > Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
 > Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
 > Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
 > Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0

That said, I can see why this return packet would be denied even if it 
were in the nat table: it would execute 'skipto 65501', which nat rule 
does not apply, as it's not outbound, and rule 65502 does not apply, as 
it's neither from AAA... nor outbound, so it's then denied by 65534.

Hope this helps.  Please cc me on any response to the list.

It would be great if someone else might care to lend an oar here; I'm 
paddling out of my depth.

cheers, Ian

[..]