Kernel NAT issues

Tue Oct 20 23:11:36 UTC 2015

Hi Ian,

Thank you very much for your response! Sorry about the late response, I have been offline for a few days.

I think I may have worked this issue out. I am bringing up a bunch of Jails today to test my firewall rules in the hopes that I have corrected my problem. I will reply back either way.

Regards,

Nathan

> On 15 Oct 2015, at 12:51 am, Ian Smith <smithi at nimnet.asn.au> wrote:
> 
> On Tue, 13 Oct 2015 13:50:04 +1000, Nathan Aherne wrote:
>> Hi Ian,
>> 
>> Thank you for your response.
>> 
>> I didnÿÿt post my ruleset because I should be able to fix the issue 
>> myself but I see now that my request to explain ÿÿhow NAT worksÿÿ was 
>> incorrect.
>> 
>> I have now included my ruleset below (as well as my initial email).
> 
> Hi Nathan,
> 
> I was really hoping someone who knows more about stateful rule handling 
> (and jail networking) might have a go at this.  Oh well I'll try, but 
> I'm a lousy mindreader, and really don't know which of the below 
> constitutes 'hairpin NAT'.  Perhaps showing your 'netstat -finet -an' 
> and 'netstat -finet -rn' may shed light on routing?  And 'ifconfig'?
> 
>> # Enable NAT
>> ipfw nat 1 config ip $jip same_ports log
> 
> I'm assuming that $jip is your WAN IP, AAA.BBB.CCC.DDD .. and that 
> WWW.XXX.YYY.ZZZ, from your posts in August, is another public IP routed 
> to you, and so traffic to it won't be subject to NAT .. correct?  But 
> the WWW... address and all 10.0/16 addresses are jails, not any separate 
> boxes you gateway for, right?  Just the one external interface, right?
> 
>> 00005 allow ip from any to any via lo0
>> 00006 deny ip from any to not me in via bce0
>> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
>> 00101 check-state
> 
> Ok, inbound from WAN is nat'd and existing stateful flows followed by 
> executing the rule that originally kept state.  Where this is a skipto, 
> skipto will be performed.  But where it's a nat rule, I've no idea .. 
> see below, but you really don't want to add keep-state (again) there.
> 
>> 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ recv bce0 keep-state
> 
> Hmm.  I'd limit this to perhaps icmptypes 0,3,8,11 - though a stateless 
> rule would make more sense especially for inbound ICMP.  But moving on ..
> 
>> 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ dst-port 65222 recv bce0 setup keep-state
> 
> Ok, but showting why plain text works better than HTML on lists :)
> 
>> 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any xmit bce0 keep-state
>> 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,80,443,22,65222 xmit bce0 setup keep-state
>> 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,123 xmit bce0 keep-state
> 
> Smells ok.
> 
>> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
>> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
> 
> Whoa, 65501 is your outbound NAT rule, albeit conditionally, and it's 
> got a problem .. see below.  These two are inbound traffic (recv) and as 
> is, skipping to 65501 will fall through two outbound rules to be denied.
> 
> Either allow them here directly, or likely better, skipto a separate
> target that then allows (or denies) them, if that's what you intended?
> 
>> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 setup keep-state
>> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
> 
> Ok, this traffic does needs to be NAT'd on the way out.
> 
>> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup keep-state
>> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup keep-state
>> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
> 
> Not clear why these tcp ports are open inbound and outbound?  Presumably 
> this is jail-to-jail traffic?  Perhaps not relevant to your problem.
> 
>> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup keep-state
>> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup keep-state
>> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
>> 65500 deny log ip from any to any
> 
> Ok.
> 
>> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
> 
> This the target for outbound traffix, xmit bce0, so nat is appropriate.  
> Does jail-to-jail traffic travels via lo1?  Or what?
> 
> This won't do anything to inbound traffic, but that really shouldn't get 
> here except returns as the result of check-state - not from 120 & 121.
> 
> But keep-state is not ok, state was already set on the skipto.  I don't 
> know how this extra keep-state might behave - does anyone have an idea?
> 
> Use 'ip4' rather than 'ip' in case this ever sees any ipv6 traffic.
> 
>> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state
> 
> So, only remaining traffic is outbound from the host itself, and traffic 
> that is to 10.0/16, but not from AAA... is to be dropped, correct?
> 
> I'm not sure whether 'allow ip .. keep-state' covers tcp, udp, icmp 
> states .. myself, I'd go for separate rules for each eg tcp, udp, .. and 
> I'd do it somewhere else than as a fall through from outbound nat rule, 
> it's confusing here, to me anyway .. unless I've missed the reason?
> 
>> 65534 deny log ip from any to any
>> 65535 deny ip from any to any
> 
> 
> Ok, now for your demo of the problem from the later mail, which I've 
> reformated to quote properly, so:
> 
>> To further illustrate my issue, this is a small log output.
>> 
>> I am running host google.com <http://google.com/> in the jail, which 
>> has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state 
>> rule.
> 
> I see you don't have logging on 101 above now.  Probably best.
> 
>> I would expect the first piece of traffic out would be UNKNOWN 
>> (does not have an entry in the state table) but it seems the 
>> returning traffic is also showing as UNKNOWN (the second 101).
> 
> I've never logged a check-state, but UNKNOWN may not mean that ..
> 
>> You can see that the traffic is returning on the same port it went 
>> out on, so its obviously the returning traffic. I am not sure why 
>> state is not being kept?
> 
> Well perhaps it is .. the return packet is from 8.8.8.8 to 10.0.0.1, so 
> it's been correctly NAT'd on the way in.  Get rid of that keep-state on 
> the nat rule at 65501 and see if not creating double entries in the 
> state table helps.  And change the skipto target on 120 & 121 to only 
> pass outbound traffic to outbound NAT rule/s.
> 
> Once you've done outbound NAT, probably best just to 'allow [log] all'?
> 
>> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
>> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
>> Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
>> Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
>> Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
>> Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
> 
> That said, I can see why this return packet would be denied even if it 
> were in the nat table: it would execute 'skipto 65501', which nat rule 
> does not apply, as it's not outbound, and rule 65502 does not apply, as 
> it's neither from AAA... nor outbound, so it's then denied by 65534.
> 
> Hope this helps.  Please cc me on any response to the list.
> 
> It would be great if someone else might care to lend an oar here; I'm 
> paddling out of my depth.
> 
> cheers, Ian
> 
> [..]