Kernel NAT issues

Tue Oct 13 05:57:59 UTC 2015

To further illustrate my issue, this is a small log output.

I am running “host google.com <http://google.com/>” in the jail, which has the IP 10.0.0.1. The UNKNOWN line is logging on the check-state rule. I would expect the first piece of traffic out would be UNKNOWN (does not have an entry in the state table) but it seems the returning traffic is also showing as UNKNOWN (the second 101). You can see that the traffic is returning on the same port it went out on, so its obviously the returning traffic. I am not sure why state is not being kept?

Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 65501 Nat UDP 10.0.0.1:57446 8.8.8.8:53 out via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 101 UNKNOWN UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 123 SkipTo 65501 UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0
Oct 13 15:50:42 host4 kernel: ipfw: 65534 Deny UDP 8.8.8.8:53 10.0.0.1:57446 in via bce0

Regards,

Nathan

> On 13 Oct 2015, at 1:50 pm, Nathan Aherne <nathan at reddog.com.au> wrote:
> 
> Hi Ian,
> 
> Thank you for your response.
> 
> I didn’t post my ruleset because I should be able to fix the issue myself but I see now that my request to explain “how NAT works” was incorrect.
> 
> I have now included my ruleset below (as well as my initial email).
> 
> # Enable NAT
> ipfw nat 1 config ip $jip same_ports log
> 
> 
> 00005 allow ip from any to any via lo0
> 00006 deny ip from any to not me in via bce0
> 00100 nat 1 log ip from any to AAA.BBB.CCC.DDD recv bce0
> 00101 check-state
> 00110 allow icmp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ recv bce0 keep-state
> 00111 allow tcp from any to WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ dst-port 65222 recv bce0 setup keep-state
> 00112 allow icmp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any xmit bce0 keep-state
> 00113 allow tcp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,80,443,22,65222 xmit bce0 setup keep-state
> 00114 allow udp from WWW.XXX.YYY <http://www.xxx.yyy/>.ZZZ to any dst-port 53,123 xmit bce0 keep-state
> 00120 skipto 65501 log tcp from any to 10.0.0.0/16 recv bce0 setup keep-state
> 00121 skipto 65501 log udp from any to 10.0.0.0/16 recv bce0 keep-state
> 00122 skipto 65501 log tcp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 setup keep-state
> 00123 skipto 65501 log udp from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
> 00200 allow log tcp from any to 10.0.0.1 dst-port 22,80,443 in setup keep-state
> 00200 allow log tcp from 10.0.0.1 to any dst-port 22,80,443 out setup keep-state
> 00200 allow log udp from 10.0.0.1 to any dst-port 53 out keep-state
> 00201 allow log tcp from any to 10.0.0.2 dst-port 22,80,443 in setup keep-state
> 00201 allow log tcp from 10.0.0.2 to any dst-port 22,80,443 out setup keep-state
> 00201 allow log udp from 10.0.0.2 to any dst-port 53 out keep-state
> 65500 deny log ip from any to any
> 65501 nat 1 log ip from 10.0.0.0/16 to not 10.0.0.0/16 xmit bce0 keep-state
> 65502 allow log ip from AAA.BBB.CCC.DDD to any xmit bce0 keep-state
> 65534 deny log ip from any to any
> 65535 deny ip from any to any
> 
> **************************************************************************************
> I sent through a question to this list a little while ago and have been trying to get IPFW NAT working since then. I have had some success but not the success I need, everything is working correctly except NAT rules for my particular use case. 
> 
> I have read every Google result on the first 50 pages when searching for “IPFW NAT” or “IPFW kernel NAT”. I would really appreciate it if someone could help me out.
> 
> My use case is as follows:
> 
> 1. I need to use hairpin NAT - I am using Jails behind a http proxy and some jails need to be able to communicate with each other but only over the WAN IP. This is why I have not use PF.
> 2. Some jails need to be able to communicate with each other on the private interface (lo1)
> 3. IPFW is configured as default deny
> 4. Each jail has a list of allowed ports for incoming and outgoing connections, these are set on the jails private IP (10.0.0.0/16)
> 5. I am using a stateful firewall.
> 
> At the moment I am testing my IPFW ruleset using “host google.com <http://google.com/> <http://google.com/ <http://google.com/>>” I can see the traffic leave the Jail, get natted, the response come back from 8.8.8.8 and the traffic is then denied. It seems like the state is not being checked or my rules are in the wrong place. I feel that I should be able to fix this but I am obviously misunderstanding is how NAT works. 
> 
> I was under the assumption that traffic flowed like this:
> 
> 1. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for public IP, the traffic is natted, it goes out the WAN interface, comes back, is natted and switched to lo1 interface, state is checked and it passes as returning traffic.
> 
> 2. Traffic comes from Jail 10.0.0.1 on lo1 interface, if traffic is for private IP, the traffic is not natted, it stays on the lo1 interface and goes directly to the 10.0.0.2 Jail.
> 
> I know I could answer my last question if “I read the code” and I have tried but am not getting it. Is my understanding of IPFW kernel NAT correct?
> 
> Regards,
> 
> Nathan
> 
> _______________________________________________
> freebsd-ipfw at freebsd.org <mailto:freebsd-ipfw at freebsd.org> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw <https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw>
> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe at freebsd.org <mailto:freebsd-ipfw-unsubscribe at freebsd.org>”
> 
> **************************************************************************************
> 
> Regards,
> 
> Nathan
> 
>> On 13 Oct 2015, at 1:37 pm, Ian Smith <smithi at nimnet.asn.au> wrote:
>> 
>> On Tue, 13 Oct 2015 12:33:52 +1000, Nathan Aherne wrote:
>> 
>>> I sent through a question to this list a little while ago and have 
>>> been trying to get IPFW NAT working since then. I have had some 
>>> success but not the success I need, everything is working correctly 
>>> except NAT rules for my particular use case.
>> 
>> Unfortunately the rest of your message failed to quote properly here, 
>> i.e not quoted indented as above, so I'll leave it out for now; perhaps 
>> it's my old mailer (pine) at fault.  Maybe plain ASCII text would help.
>> 
>> That said, without sharing your actual ruleset with us, sanitised if 
>> need be, it seems unlikely that anyone will be able to work out what 
>> might be happening here solely from your textual description.
>> 
>> cheers, Ian
> 
> _______________________________________________
> freebsd-ipfw at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-ipfw
> To unsubscribe, send any mail to "freebsd-ipfw-unsubscribe at freebsd.org"