Kernel NAT issues
Julian Elischer
julian at freebsd.org
Sat Nov 21 14:08:50 UTC 2015
I note, no answers yet..
I plan on spending some time tomorrow trying to understand this issue.
just letting you know..
On 21/11/2015 10:06 AM, Nathan Aherne wrote:
> I had a bit of a think about how to describe what I am trying to
> achieve.
>
> I am treating each jail likes its own little "virtual machine”. The
> jail provides certain services, using things like nginx or nodejs,
> php-fpm, mysql or postgresql. The jails can control connections to
> themselves by configuring the firewall ports that are opened on the
> IP their IP (10.0.0.0/16 or a public IP). I know the jails have no
> firewall of their own, the firewall is configured from the host.
>
> I want each jail or “virtual machine” to be able to communicate with
> one another and the wider internet. When a jail does a DNS query for
> another App jail, it may get a public IP on its own Host (or it may
> get another host) and it has no issues being able to communicate
> with another jail on the same host.
>
> At the moment all of the above is working perfectly except for jail
> to jail communication on the same host (when the communication is
> not directly between 10.0.0.0/16 IP addresses).
>
> Regards,
>
> Nathan
>
>> On 21 Nov 2015, at 9:12 am, Nathan Aherne <nathan at reddog.com.au
>> <mailto:nathan at reddog.com.au>> wrote:
>>
>> I am not exactly sure how to draw the setup so it doesn’t confuse
>> the situation. The setup is extremely simple (I am not running
>> vimage), jails running on the 10.0.0.0/16 (cloned lo1 interface)
>> network or with public IPs. The jails with private IPs are the HTTP
>> app jails. The Host runs a HTTP Proxy (nginx) and forwards traffic
>> to each HTTP App jail based on the URL it receives. The jails with
>> public IPs are things like database jails which cannot be proxied
>> by the Host.
>>
>> I can happily communicate with any jail from my laptop (externally)
>> but when I want one jail to communicate with another jail (for
>> example an App Jail communicating with the database jail) the
>> traffic shows as backwards (destination:port -> source:port) in the
>> IPFW logs (tshark shows the traffic correctly source:port ->
>> destination:port). The jail to jail traffic tries to go over the
>> lo1 interface (backwards) and is blocked. Below is some IPFW logs
>> of an App jail (10.0.0.25) communicating with the database jail
>> (aaa.bbb.ccc.ddd)
>>
>> IPFW logs. The lines labelled UNKNOWN is the check-state rule
>> (everything is labelled UNKNOWN even if it is KNOWN traffic)
>>
>> Nov 21 08:49:07 host5 kernel: ipfw: 101 UNKNOWN TCP
>> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
>> Nov 21 08:49:07 host5 kernel: ipfw: 65501 Deny TCP
>> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
>> Nov 21 08:49:10 host5 kernel: ipfw: 101 UNKNOWN TCP
>> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
>> Nov 21 08:49:10 host5 kernel: ipfw: 65501 Deny TCP
>> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
>> Nov 21 08:49:13 host5 kernel: ipfw: 101 UNKNOWN TCP
>> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
>> Nov 21 08:49:13 host5 kernel: ipfw: 65501 Deny TCP
>> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
>> Nov 21 08:49:16 host5 kernel: ipfw: 101 UNKNOWN TCP
>> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
>> Nov 21 08:49:16 host5 kernel: ipfw: 65501 Deny TCP
>> eee.fff.gg.hhh:5432 10.0.0.25:42957 out via lo1
>>
>> tshark output (loopback and wan interface capture for port 5432)
>>
>> Capturing on 'Loopback' and 'bce0'
>> 1 0.000000 10.0.0.25 -> eee.fff.gg.hhh TCP 64 42957→5432 [SYN]
>> Seq=0 Win=65535 Len=0 MSS=16344 WS=64 SACK_PERM=1 TSval=142885525
>> TSecr=0
>> 2 3.013905 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP
>> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344
>> WS=64 SACK_PERM=1 TSval=142888539 TSecr=0
>> 3 6.241658 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP
>> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344
>> WS=64 SACK_PERM=1 TSval=142891767 TSecr=0
>> 4 9.451516 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP
>> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344
>> WS=64 SACK_PERM=1 TSval=142894976 TSecr=0
>> 5 12.654656 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP
>> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344
>> WS=64 SACK_PERM=1 TSval=142898180 TSecr=0
>> 6 15.863900 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP
>> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344
>> WS=64 SACK_PERM=1 TSval=142901389 TSecr=0
>> 7 22.076655 10.0.0.25 -> eee.fff.gg.hhh TCP 64 [TCP
>> Retransmission] 42957→5432 [SYN] Seq=0 Win=65535 Len=0 MSS=16344
>> WS=64 SACK_PERM=1 TSval=142907602 TSecr=0
>>
>>
>>> If so, what sort of routing is setup on both host and jails?
>>
>> Routing is what would be added by default (whatever the host system
>> adds when adding an IP), there is no custom routing. I have
>> wondered if I need to modify the routing table to get this to work.
>>
>> Below is the output of netstat -rn
>>
>> www.xxx.yy <http://www.xxx.yy/>.zzz is the gateway address
>> eee.fff.gg.hhh is the database jail public IP
>> aaa.bbb.cc.ddd is the public IP for NAT
>> lll.mmm.nn.ooo is the Hosts public IP
>>
>>
>> Routing tables
>>
>> Internet:
>> Destination Gateway Flags Netif Expire
>> default www.xxx.yy <http://www.xxx.yy/>.zzz UGS bce0
>> 10.0.0.1 link#6 UH lo1
>> 10.0.0.2 link#6 UH lo1
>> 10.0.0.3 link#6 UH lo1
>> 10.0.0.4 link#6 UH lo1
>> 10.0.0.5 link#6 UH lo1
>> 10.0.0.6 link#6 UH lo1
>> 10.0.0.7 link#6 UH lo1
>> 10.0.0.8 link#6 UH lo1
>> 10.0.0.9 link#6 UH lo1
>> 10.0.0.10 link#6 UH lo1
>> 10.0.0.11 link#6 UH lo1
>> 10.0.0.12 link#6 UH lo1
>> 10.0.0.13 link#6 UH lo1
>> 10.0.0.14 link#6 UH lo1
>> 10.0.0.15 link#6 UH lo1
>> 10.0.0.16 link#6 UH lo1
>> 10.0.0.17 link#6 UH lo1
>> 10.0.0.18 link#6 UH lo1
>> 10.0.0.19 link#6 UH lo1
>> 10.0.0.20 link#6 UH lo1
>> 10.0.0.21 link#6 UH lo1
>> 10.0.0.22 link#6 UH lo1
>> 10.0.0.23 link#6 UH lo1
>> 10.0.0.24 link#6 UH lo1
>> 10.0.0.25 link#6 UH lo1
>> 10.0.0.26 link#6 UH lo1
>> www.xxx.yy.zzz/25 <http://www.xxx.yy.zzz/25> link#1 U
>> bce0
>> eee.fff.gg.hhh link#1 UHS lo0
>> eee.fff.gg.hhh/32 link#1 U bce0
>> aaa.bbb.cc <http://aaa.bbb.cc/>.ddd link#1 UHS lo0
>> aaa.bbb.cc.ddd/32 link#1 U bce0
>> lll.mmm.nn.ooo link#1 UHS lo0
>> 127.0.0.1 link#5 UH lo0
>>
>> Internet6:
>> Destination Gateway Flags
>> Netif Expire
>> ::/96 ::1 UGRS lo0
>> ::1 link#5 UH lo0
>> ::ffff:0.0.0.0/96 ::1 UGRS
>> lo0
>> fe80::/10 ::1 UGRS lo0
>> fe80::%lo0/64 link#5 U lo0
>> fe80::1%lo0 link#5 UHS lo0
>> ff01::%lo0/32 ::1 U lo0
>> ff02::/16 ::1 UGRS lo0
>> ff02::%lo0/32 ::1 U lo0
>>
>>> Anything like ?
>>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search
>>
>> Yes just like that.
>>
>> Regards,
>>
>> Nathan
>>
>>> On 19 Nov 2015, at 2:46 am, Ian Smith <smithi at nimnet.asn.au
>>> <mailto:smithi at nimnet.asn.au>> wrote:
>>>
>>> On Wed, 18 Nov 2015 22:17:29 +0800, Julian Elischer wrote:
>>>> On 11/18/15 8:40 AM, Nathan Aherne wrote:
>>>>> For some reason hairpin (loopback nat or nat reflection) does
>>>>> not seem to
>>>>> be working, which is why I chose IPFW in the first place.
>>>
>>>> it would be good to see a diagram of what this actually means.
>>>
>>> Anything like ?
>>> http://kb.juniper.net/InfoCenter/index?page=content&id=KB24639&actp=search
>>>
>>> Was this so one jail can only access service/s provided by other
>>> jail/s,
>>> both/all with internal NAT'd addresses, by using only the public
>>> address
>>> and port of the 'router', which IIRC this is a single system with
>>> jails?
>>>
>>> If so, what sort of routing is setup on both host and jails?
>>>
>>> (blindfolded, no idea where I've pinned the donkey's tail :)
>>>
>>> cheers, Ian
>>
>
More information about the freebsd-ipfw
mailing list