Jails routing and localhost

[Isaac (.ike) Levy]

Hi Ole,

I cannot comment on ezjail specifics, but can add notes on how jails fundamentally handle localhost, which may help you,

On Thu, Jan 18, 2018, at 7:23 AM, Ole wrote:
> Hi,
> 
> I have some questions about how routing works for jails. 
> 
> I have a FreeBSD 11.1 host in a datacenter. Which has only a routed IP
> and different /29 routed networks. The IP is setup as /32 and there is a
> default route to the router of the datacenter:
> 
> 
>   #ifconfig em1
>     (...)
>     inet a.a.a.57 netmask 0xffffffff broadcast a.a.a.57
>     (...)
> 
> 
>   # netstat -rn
>     (...)
>     Destination        Gateway            Flags     Netif Expire
>     default            a.a.a.1            UGS         em1
>     (...)
> 
> 
> If I create jails like
> 
>   # ezjail-admin create somejail 'lo1|b.b.b.238,lo1|127.b.b.238'
> 
> everything is fine until some service in the jail tries to bind to
> 127.0.0.1. Because it will bind to the public IP b.b.b.238.
> The Handbook [1] tells 
> 
>   "Inside a jail, access to the loopback address 127.0.0.1 is
>   redirected to the first IP address assigned to the jail."

Right- so if you don't assign a loopback address at all, loopback will use to the first IP assigned the host (another response in this thread).

Because your hardware host has the 127.0.0.1 address, if you were to assign it to jails, that would mean all jails would all be communicating using it, which would be bad - (I'm not sure if it even works or if jail(2) prevents it from working).

RFC 3330 tells us, http://www.ietf.org/rfc/rfc3330.txt

   127.0.0.0/8 - This block is assigned for use as the Internet host
   loopback address.  A datagram sent by a higher level protocol to an
   address anywhere within this block should loop back inside the host.
   This is ordinarily implemented using only 127.0.0.1/32 for loopback,
   but no addresses within this block should ever appear on any network
   anywhere [RFC1700, page 5].

So, here's what I've been doing for many years:

- Add another 127.x.x.x address to lo0 on your hardware host  (I typically match the last octects to the public address I'm using, just my way of keeping track of things), for example, this IPv4 address could look like: "127.4.4.4/32".  You can add these single /32 addresses right to the lo0 interface on the host machine.

- Start your jail, first binding "127.4.4.4/32" to it as one of the IP interfaces- follow with your other IP's.

- In your jail, edit /etc/resolv.conf so that the 'localhost' entry, matches your IP above, "127.4.4.4".

Viola- you now have localhost!

--
This process is quite counter-intuitive,  since who ever really thinks about 127.0.0.0/8 as an actual netblock?  (Since these addresses "no addresses within this block should ever appear on any network anywhere [RFC1700, page 5]."

The same principle applies to IPv6 localhost in jails.

> 
> If I change the order of the IP-Adresses the service will bind to
> 127.b.b.238. But inside the Jail Networking fails in a way that I can't
> debug. I can conntect from the outside via ssh but I can't connect from
> the Jail to an external Server. I can't find any differences in
> routing table or ifconfig between both setups.

>From what you wrote above, I agree with the other person who responded- it may be the order of when you specify interfaces, (or how ezjail does).  Or, it may be that you're not making the localhost address a /32 to isolate it.

--
One more caveat: bad software :)
I've seen plenty of fine software which follows very bad form and hardcodes 127.0.0.1, instead of calling 'localhost' for various operations.  Simple answer here: file a bug and point to internet RFC's if 3rd party software, or, go have a chat with your colleagues if the software is in-house.

> 
> 
> I also tried to use tap interfaces instead of lo, but it results in the
> same. 

  (From a practical security perspective, I've wondered for years if making abstracted interfaces for each localhost in each jail had any advantages, but that's a tangent here.)

> 
> I wonder how others solve this problem. I searched a lot, but couldn't
> find a solution. Maybe you don't have a solution, but can give me a
> hint to debug the Problem. Thank you!

Hope this helps, tell us how it goes!

Best,
.ike



> 
> 
> regards
> Ole
> 
> [1] https://www.freebsd.org/doc/handbook/jails-ezjail.html
> Email had 1 attachment:
> + Attachment2
>   1k (application/pgp-signature)