Tutorial for Hierarchical Jails?

[Jamie Gritton]

Without going into the current rc system, which isn't up to the task
of hierarchical jails, here's a minimal set of parameters/commands to
create hierarchical jails that can still ping:

# jail -c name=foo host.hostname=foo allow.raw_sockets children.max=99 
ip4.addr=10.20.12.68 persist
# jexec foo /bin/csh
foo# jail -c name=bar host.hostname=bar allow.raw_sockets 
ip4.addr=10.20.12.68 persist
foo# jexec bar /bin/csh
bar# ping gritton.org
PING gritton.org (161.58.222.4): 56 data bytes
64 bytes from 161.58.222.4: icmp_seq=0 ttl=54 time=78.344 ms

- Jamie


Edwin Shao wrote:
> The base system has allow_raw_sockets, the first level jail also has 
> allow_raw_sockets and has the exact same configuration as the base 
> system (I use puppet to manage config files.) I can't set 
> allow_raw_sockets anyway for the second-level jail without manually 
> invoking the jail command.
> 
> On Tue, Sep 29, 2009 at 7:08 AM, Jamie Gritton <jamie at freebsd.org 
> <mailto:jamie at freebsd.org>> wrote:
> 
>     Does the base system have security.jail.allow_raw_sockets=1? You need to
>     have that, or set the jail's allow.raw_sockets. You can't set the jail's
>     permissions from within the jail itself. If you have multiple jail
>     levels, then both jails need to allow raw sockets - a jail can't allow a
>     child jail to do what it can't do itself.
> 
>     - Jamie
> 
> 
>     Edwin Shao wrote:
> 
>         One other thing that is odd: hierarchical jails don't seem to
>         inherit some sysctls such as allow_raw_socket.
> 
>         In the host (jail), rc.conf has jail_set_allow_raw_sockets="YES"
>         and sysctl.conf has "security.jail.allow_raw_sockets=1", but no
>         child jail can ping out:
>         neko# ping google.com <http://google.com> <http://google.com>
> 
>         ping: socket: Operation not permitted
> 
>         What is happening in this case?
>         Thank you for your time again.
> 
> 
>         On Tue, Sep 29, 2009 at 12:16 AM, Jamie Gritton
>         <jamie at freebsd.org <mailto:jamie at freebsd.org>
>         <mailto:jamie at freebsd.org <mailto:jamie at freebsd.org>>> wrote:
> 
>            The sysctls not only don't get written to, they don't have
>         any useful
>            information to read either. They only describe the existence
>         and format
>            of the various jail parameters. Sorry, but there;s no way to
>         set a
>            default children.max parameter or inherit it from the parent.
>         We've
>            decided to set the default to the most secure/restrictive in
>         many cases.
>            Once we've come up with a new jail configuration interface,
>         this won't
>            be such a hassle.
> 
>            The devfs errors are probably something that will have to be
>         addressed
>            in a later revision - I haven't looked in the devfs direction
>         so I'm not
>            sure about that. The mount error may be related to the first
>         jail's
>            allow.mount parameter (whose default comes from
>            security.jail.mount_allowed).
> 
>            - Jamie
> 
>            Edwin Shao wrote:
> 
>                Thanks, that worked for me.
> 
>                * Using jail to change children.max on the parent does not
>                affect `sysctl security.jail.param.children.max` in the
>         child.
>                 Also security.jail.param.children.cur never changes
>         either. Not
>                sure if that's intended behavior.
>                * Is there any way to persist the
>                security.jail.param.children.max parameter without
>         entering the
>                jail command every time? * I get the following output when I
>                create a jail inside a jail:
> 
>                hyper ~> ezjail-admin start neko
>                Configuring jails:.
>                Starting jails:devfs rule: ioctl DEVFSIO_RGETNEXT:
>         Operation not
>                permitted
>                devfs rule: ioctl DEVFSIO_RGETNEXT: Operation not permitted
>                /etc/rc.d/jail: WARNING: devfs_set_ruleset: you must
>         specify a
>                ruleset number
>                devfs rule: ioctl DEVFSIO_SAPPLY: Operation not permitted
>                ln: log: Operation not permitted
>                mount: proc : Operation not permitted
>                 neko.
> 
>                I'm using the same configuration values as in the
>         parent's jail,
>                which work. Everything seems to work alright inside the
>         jail, so
>                I assume the errors are safe to ignore?
> 
>                Thanks again!
>                - Edwin
> 
>                On Mon, Sep 28, 2009 at 9:11 PM, Bjoern A. Zeeb
>                <bzeeb-lists at lists.zabbadoz.net
>         <mailto:bzeeb-lists at lists.zabbadoz.net>
>                <mailto:bzeeb-lists at lists.zabbadoz.net
>         <mailto:bzeeb-lists at lists.zabbadoz.net>>
>                <mailto:bzeeb-lists at lists.zabbadoz.net
>         <mailto:bzeeb-lists at lists.zabbadoz.net>
>                <mailto:bzeeb-lists at lists.zabbadoz.net
>         <mailto:bzeeb-lists at lists.zabbadoz.net>>>> wrote:
> 
>                   On Mon, 28 Sep 2009, Edwin Shao wrote:
> 
>                       Hi Jamie,
>                       When I try to change the parameter, nothing happens:
>                       rescue /etc> sudo sysctl
>         security.jail.param.children.max=1
>                       security.jail.param.children.max: 0 -> 0
> 
>                       rescue /etc> sudo sysctl
>         security.jail.param.children.max
>                       security.jail.param.children.max: 0
> 
>                       Am I doing this incorrectly?
> 
> 
>                   Yes. It's a parameter to jail(8).  The security.jail.param
>                sysctls can
>                   be seen as a list of possible options valid to
>         jail(8).  See
>                man 8 jail
>                   for the exact details.
> 
>                   /bz
> 
>                   --    Bjoern A. Zeeb           What was I talking
>         about and
>                who are you again?
> 
> 
> 
>