Jails and networks

Thu Aug 23 18:45:06 UTC 2018

Greetings.

I'm having difficulty creating a jail which is able to see the outside
world.  The various recipes I've found seem to be subtly contradictory:
I'm trying to understand what they're doing rather than dumbly following
them, and my lack of success here is telling me that my mental model of
jails+networking doesn't quite match reality.  I think I'm on the verge
of a very educational experience....

I'm using ezjail, on 11.2.

Sources:

   * The manual [1] describes basic usage, but mentions release 9.3; I
get the impression that ezjail's procedure for starting and configuring
jails (using /etc/jail.conf rather than the old 4 arguments) is slightly
but significantly incompatible with 11.2.

   * The ezjail documentation [2] describes setting up a jail using
em0|10.0.0.2, very straightforwardly

   * A forum post [3] describes setting up a jail using ezjail and pf.
Now, I don't think I need pf in my situation, so I want to skip that
part of the instructions.  But I now suspect I'm doing so naively.

   * Another forum post [4] describes setting up both a VIMAGE and a
non-VIMAGE jail, and is usefully explicit about the contents of the
/etc/jail.conf file.  This is the one I've been following most closely,
but I realise that I don't understand why it configures a bridge
interface, but adds only a single real interface igb0 to it (my model of
a bridge interface is that it necessarily involves two interfaces, or
does the igb0 in the host and the one in the client count as two?).

My host is on a 172.16.0.0/12 private network, which is routable
locally, though it has to use a proxy to get to the web.  I want to set
up a jail on (slightly at random) 192.168.11.128.

I have:

   * net.inet.ip.forwarding: 1
   * igb0 configured with the correct IP address and mask, not aliased
at all
   * I've created lo1

My /etc/jail.conf looks like

     exec.start = "/bin/sh /etc/rc";
     exec.stop = "/bin/sh /etc/rc.shutdown";
     exec.clean;

     path = "/local/jails/$name";

     mount.fstab = "/etc/jail/fstab.${name}";
     mount.devfs;
     mount.fdescfs;
     mount.procfs;

     host.hostname = "${name}.local";

     devfs_ruleset         = "4";

     norman {
         # test jail
         ip4.addr = "192.168.11.128";
         interface = "igb0";
     }

and the non-comment lines in /usr/local/etc/ezjail.conf look like

     ezjail_jaildir=/local/jails
     ezjail_ftphost=http://ftp.uk.freebsd.org
     ezjail_use_zfs="YES"
     ezjail_use_zfs_for_jails="YES"
     ezjail_jailzfs=zroot/local/jails

I've created a ezjail flavour called 'norman' (with the inevitable
solipsism).

My _understanding_ is that this sets the jail to use the igb0 interface
in the host (a non-VIMAGE jail doesn't have a separate networking
stack).

I create the jail

     ezjail-admin create -f norman -c zfs norman
'lo1|127.0.1.1,igb0|192.168.11.128'

lo1 first, as suggested in [1].  My impression is that that sets up the
loopback interface within the jail to be an alias of lo0 in the host,
and attaches 192.168.11.128 to igb0 in the  jail.

Then I start the jail

     jail -c norman

it starts up sshd promptly, but takes a long time (presumably timing out
in fact) to start sendmail_submit and sendmail_msp_queue.  Then

     jexec 4 /bin/sh

lets me see

# cat /etc/resolv.conf
search physics.gla.ac.uk
nameserver 130.209.4.16
nameserver 130.209.4.18
# ifconfig igb0
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
1500
   options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
  ether a4:bf:01:26:7d:b1
  hwaddr a4:bf:01:26:7d:b1
  inet 192.168.11.128 netmask 0xffffffff broadcast 192.168.11.128
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active

...which looks right.  But

# host www.gla.ac.uk
;; connection timed out; no servers could be reached
#

The routing table is very simple:

# netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
192.168.11.128     link#3             UHS         lo0

I don't think I've done anything at all exotic here, and the resolv.conf
contents and ifconfig output looks as I'd expect.  The routing table
doesn't have a default route, but (a) if this interface is just the same
as the same-named one in the host, so ... *mumble*; and (b) the various
recipes I've quoted don't anywhere mention having to add a default
route, so I don't think that can be what I'm missing.

I'm wondering if there's something to do with the private network the
host is on.  But that can talk to the network without difficulty, and in
any case http_proxy is correctly set in the jail.

I've seen a mention of epair(4), but I don't think that's relevant.

So I'm clearly misunderstanding something terribly important (and
embarrassingly obvious in retrospect), which hasn't magically become
clear by my explaining the steps clearly to myself here.  I suspect I
don't _actually_ understand the relationship between the jail's
interfaces and the host's -- they seem the same but not the same in some
very uncomfortable way.

Any epiphanies gratefully received.

Best wishes,

Norman

[1] https://www.freebsd.org/doc/handbook/jails-ezjail.html
[2] https://erdgeist.org/arts/software/ezjail/
[3] https://forums.freebsd.org/threads/30063/
[4] https://forums.freebsd.org/threads/49561/

--
Norman Gray  :  https://nxg.me.uk
SUPA School of Physics and Astronomy, University of Glasgow, UK

[University of Glasgow: The Times Scottish University of the Year 2018]