git: 612b7cc172 - main - Handbook - Jails: Rewrite the chapter
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Wed, 20 Sep 2023 08:33:43 UTC
The branch main has been updated by carlavilla:
URL: https://cgit.FreeBSD.org/doc/commit/?id=612b7cc1721224c494c5b2600188e1508bb5611b
commit 612b7cc1721224c494c5b2600188e1508bb5611b
Author: Sergio Carlavilla Delgado <carlavilla@FreeBSD.org>
AuthorDate: 2023-09-20 08:15:40 +0000
Commit: Sergio Carlavilla Delgado <carlavilla@FreeBSD.org>
CommitDate: 2023-09-20 08:15:40 +0000
Handbook - Jails: Rewrite the chapter
Changes:
- Add jail types info: Thick, Thin, VNET and Linux Jails
- Add info about the necessary configuration on the host system
- Add info about types of networking in the jails
- Explain the jail config file
- Add info about how to configure Thick, Thin, VNET and Linux Jails
- Explain how to upgrade the different types of jails
- Add info about the resources limits in a Jail
- Add info about different jail managers and containers
Information obtained from:
- https://man.freebsd.org/cgi/man.cgi?query=jail&sektion=8
- https://wiki.freebsd.org/Jails
- https://man.freebsd.org/cgi/man.cgi?rctl(8)
- https://vermaden.wordpress.com/2023/06/28/freebsd-jails-containers/
- https://clinta.github.io/freebsd-jails-the-hard-way/
- https://jacob.ludriks.com/2017/06/07/FreeBSD-Thin-Jails/
- Jails and VNET a guide - Derik J. Ramirez
- https://wb-hk.blogspot.com/2016/04/freebsd-jails-4-thin-jails-using-nullfs.html
- https://blog.uidrafter.com/freebsd-jails-network-setup
- https://forums.freebsd.org/threads/setting-up-a-debian-linux-jail-on-freebsd.68434/
- https://github.com/msimerson/Mail-Toaster-6/wiki/Linux-Jails
- https://etherealwake.com/2021/08/freebsd-jail-networking/
- https://wiki.freebsd.org/TomMarcoen/JailNetworking
- https://weblog.antranigv.am/posts/2020/06/vnet-jail-howto/
PR: 203641, 200905, 209157, 248150, 263330, 264317,
226282, 239861, 166358, 178221, 265410
Reviewed by: karels@, fernape@, jrm@
Differential Revision: https://reviews.freebsd.org/D41450
---
.../content/en/books/handbook/jails/_index.adoc | 1492 ++++++++++----------
1 file changed, 715 insertions(+), 777 deletions(-)
diff --git a/documentation/content/en/books/handbook/jails/_index.adoc b/documentation/content/en/books/handbook/jails/_index.adoc
index 2a51f99432..36b00fd200 100644
--- a/documentation/content/en/books/handbook/jails/_index.adoc
+++ b/documentation/content/en/books/handbook/jails/_index.adoc
@@ -1,17 +1,17 @@
---
-title: Chapter 17. Jails
+title: Chapter 17. Jails and Containers
part: Part III. System Administration
prev: books/handbook/security
next: books/handbook/mac
description: Jails improve on the concept of the traditional chroot environment in several ways
-tags: ["jails", "creating", "managing", "updating", "ezjail"]
+tags: ["jails", "creating", "managing", "updating"]
showBookMenu: true
weight: 21
path: "/books/handbook/"
---
[[jails]]
-= Jails
+= Jails and Containers
:doctype: book
:toc: macro
:toclevels: 1
@@ -48,6 +48,24 @@ toc::[]
include::../../../../../shared/asciidoctor.adoc[]
endif::[]
+// Related bugs to this chapter: 203641, 200905, 209157, 248150, 263330, 264317, 226282, 239861, 166358, 178221, 265410
+// Documentation checked
+// https://man.freebsd.org/cgi/man.cgi?query=jail&sektion=8
+// https://wiki.freebsd.org/Jails
+// https://man.freebsd.org/cgi/man.cgi?rctl(8)
+// https://vermaden.wordpress.com/2023/06/28/freebsd-jails-containers/
+// https://clinta.github.io/freebsd-jails-the-hard-way/
+// https://marcocetica.com/posts/understanding-freebsd-jail/
+// https://jacob.ludriks.com/2017/06/07/FreeBSD-Thin-Jails/
+// Jails and VNET a guide - Derik J. Ramirez
+// https://wb-hk.blogspot.com/2016/04/freebsd-jails-4-thin-jails-using-nullfs.html
+// https://blog.uidrafter.com/freebsd-jails-network-setup
+// https://forums.freebsd.org/threads/setting-up-a-debian-linux-jail-on-freebsd.68434/
+// https://github.com/msimerson/Mail-Toaster-6/wiki/Linux-Jails
+// https://etherealwake.com/2021/08/freebsd-jail-networking/
+// https://wiki.freebsd.org/TomMarcoen/JailNetworking
+// https://weblog.antranigv.am/posts/2020/06/vnet-jail-howto/
+
[[jails-synopsis]]
== Synopsis
@@ -60,1171 +78,1091 @@ Jails build upon the man:chroot[2] concept, which is used to change the root dir
This creates a safe environment, separate from the rest of the system.
Processes created in the chrooted environment can not access files or resources outside of it.
For that reason, compromising a service running in a chrooted environment should not allow the attacker to compromise the entire system.
+
However, a chroot has several limitations.
It is suited to easy tasks which do not require much flexibility or complex, advanced features.
Over time, many ways have been found to escape from a chrooted environment, making it a less than ideal solution for securing services.
Jails improve on the concept of the traditional chroot environment in several ways.
+
In a traditional chroot environment, processes are only limited in the part of the file system they can access.
The rest of the system resources, system users, running processes, and the networking subsystem are shared by the chrooted processes and the processes of the host system.
Jails expand this model by virtualizing access to the file system, the set of users, and the networking subsystem.
More fine-grained controls are available for tuning the access of a jailed environment.
Jails can be considered as a type of operating system-level virtualization.
-A jail is characterized by four elements:
+This chapter covers:
-* A directory subtree: the starting point from which a jail is entered. Once inside the jail, a process is not permitted to escape outside of this subtree.
-* A hostname: which will be used by the jail.
-* An IP address: which is assigned to the jail. The IP address of a jail is often an alias address for an existing network interface.
-* A command: the path name of an executable to run inside the jail. The path is relative to the root directory of the jail environment.
+* What a jail is and what purpose it may serve in FreeBSD installations.
+* The different type of jails.
+* The different way to configure the network for a jail.
+* The jail configuration file.
+* How to create the different types of jails.
+* How to start, stop and restart a jail.
+* The basics of jail administration, both from inside and outside the jail.
+* How to upgrade the different types of jails.
+* A incomplete list of the different FreeBSD jail managers.
-Jails have their own set of users and their own `root` account which are limited to the jail environment.
-The `root` account of a jail is not allowed to perform operations to the system outside of the associated jail environment.
+[[jail-types]]
+== Jail types
-This chapter provides an overview of the terminology and commands for managing FreeBSD jails.
-Jails are a powerful tool for both system administrators, and advanced users.
+Some administrators divide jails into different types, although the underlying technology is the same.
+Each administrator will have to assess what type of jail to create in each case depending on the problem they have to solve.
-After reading this chapter, you will know:
+Below can be found a list of the different types, their characteristics, etc.
-* What a jail is and what purpose it may serve in FreeBSD installations.
-* How to build, start, and stop a jail.
-* The basics of jail administration, both from inside and outside the jail.
+[[thick-jails]]
+=== Thick Jails
-[IMPORTANT]
-====
-Jails are a powerful tool, but they are not a security panacea.
-While it is not possible for a jailed process to break out on its own, there are several ways in which an unprivileged user outside the jail can cooperate with a privileged user inside the jail to obtain elevated privileges in the host environment.
+A thick jail is a more traditional form of FreeBSD Jail.
+In a thick jail, a complete copy of the base system is replicated within the jail's environment.
+This means that the jail has its own separate instance of the FreeBSD base system, including libraries, executables, and configuration files.
+The jail can be thought of as an almost complete standalone FreeBSD installation, but running within the confines of the host system.
+This isolation ensures that the processes within the jail are kept separate from those on the host and other jails.
-Most of these attacks can be mitigated by ensuring that the jail root is not accessible to unprivileged users in the host environment.
-As a general rule, untrusted users with privileged access to a jail should not be given access to the host environment.
-====
+Advantages of Thick Jails:
-[[jails-terms]]
-== Terms Related to Jails
+* High degree of isolation: Processes within the jail are isolated from the host system and other jails.
+* Independence: Thick jails can have different versions of libraries, configurations, and software than the host system or other jails.
+* Security: Since the jail contains its own base system, vulnerabilities or issues affecting the jail environment won't directly impact the host or other jails.
-To facilitate better understanding of parts of the FreeBSD system related to jails, their internals and the way they interact with the rest of FreeBSD, the following terms are used further in this chapter:
+Disadvantages of Thick Jails:
-man:chroot[8] (command)::
-Utility, which uses man:chroot[2] FreeBSD system call to change the root directory of a process and all its descendants.
+* Resource overhead: Because each jail maintains its own separate base system, it consumes more resources compared to thin jails.
+* Maintenance: Each jail requires its own maintenance and updates for its base system components.
-man:chroot[2] (environment)::
-The environment of processes running in a "chroot".
-This includes resources such as the part of the file system which is visible, user and group IDs which are available, network interfaces and other IPC mechanisms, etc.
+[[thin-jails]]
+=== Thin Jails
-man:jail[8] (command)::
-The system administration utility which allows launching of processes within a jail environment.
+A thin jail shares the base system using OpenZFS snapshots or NullFS mounts from a template.
+Only a minimal subset of base system is duplicated for each thin jail, resulting in less resource consumption compared to a thick jail.
+However, this also means that thin jails have less isolation and independence compared to thick jails.
+Changes in shared components could potentially affect multiple thin jails simultaneously.
-host (system, process, user, etc.)::
-The controlling system of a jail environment.
-The host system has access to all the hardware resources available, and can control processes both outside of and inside a jail environment.
-One of the important differences of the host system from a jail is that the limitations which apply to superuser processes inside a jail are not enforced for processes of the host system.
+In summary, a FreeBSD Thick Jail is a type of FreeBSD Jail that replicates a substantial portion of the base system within the isolated environment.
-hosted (system, process, user, etc.)::
-A process, user or other entity, whose access to resources is restricted by a FreeBSD jail.
+Advantages of Thin Jails:
-[[jails-build]]
-== Creating and Controlling Jails
+* Resource Efficiency: Thin jails are more resource-efficient compared to thick jails. Since they share most of the base system, they consume less disk space and memory. This makes it possible to run more jails on the same hardware without consuming excessive resources.
+* Faster Deployment: Creating and launching thin jails is generally faster compared to thick jails. This can be particularly advantageous when you need to rapidly deploy multiple instances.
+* Unified Maintenance: Since thin jails share the majority of their base system with the host system, updates and maintenance of common base system components (such as libraries and binaries) only need to be done once on the host. This simplifies the maintenance process compared to maintaining individual base system for each thick jail.
+* Shared Resources: Thin jails can more easily share common resources such as libraries and binaries with the host system. This can potentially lead to more efficient disk caching and improved performance for applications within the jail.
-Some administrators divide jails into the following two types: "complete" jails, which resemble a real FreeBSD system, and "service" jails, dedicated to one application or service, possibly running with privileges.
-This is only a conceptual division and the process of building a jail is not affected by it.
-When creating a "complete" jail there are two options for the source of the userland: use prebuilt binaries (such as those supplied on an install media) or build from source.
+Disadvantages of Thin Jails:
-=== Installing a Jail
+* Reduced Isolation: The primary disadvantage of thin jails is that they offer less isolation compared to thick jails. Since they share a significant portion of the template's base system, vulnerabilities or issues affecting shared components could potentially impact multiple jails simultaneously.
+* Security Concerns: The reduced isolation in thin jails could pose security risks, as a compromise in one jail might have a greater potential to affect other jails or the host system.
+* Dependency Conflicts: If multiple thin jails require different versions of the same libraries or software, managing dependencies can become complex. In some cases, this might require additional effort to ensure compatibility.
+* Compatibility Challenges: Applications within a thin jail might encounter compatibility issues if they assume a certain base system environment that differs from the shared components provided by the template.
-[[jails-install-internet]]
-==== To install a Jail from the Internet
+[[vnet-jails]]
+=== VNET Jails
-The man:bsdinstall[8] tool can be used to fetch and install the binaries needed for a jail.
-This will walk through the picking of a mirror, which distributions will be installed into the destination directory, and some basic configuration of the jail:
+A FreeBSD VNET jail is a virtualized environment that allows for the isolation and control of network resources for processes running within it.
+It provides a high level of network segmentation and security by creating a separate network stack for processes within the jail,
+ensuring that network traffic within the jail is isolated from the host system and other jails.
-[source,shell]
-....
-# bsdinstall jail /here/is/the/jail
-....
+In essence, FreeBSD VNET jails add a network configuration mechanism.
+This means a VNET jail can be created as a Thick or Thin Jail.
-Once the command is complete, the next step is configuring the host to run the jail.
+[[linux-jails]]
+=== Linux Jails
-[[jails-install-iso]]
-==== To install a Jail from an ISO
+A FreeBSD Linux Jail is a feature in the FreeBSD operating system that enables the use of Linux binaries and applications within a FreeBSD jail.
+This functionality is achieved by incorporating a compatibility layer that allows certain Linux system calls and libraries to be translated and executed on the FreeBSD kernel.
+The purpose of a Linux Jail is to facilitate the execution of Linux software on a FreeBSD system without needing a separate Linux virtual machine or environment.
-To install the userland from installation media, first create the root directory for the jail.
-This can be done by setting the `DESTDIR` variable to the proper location.
+[[host-configuration]]
+== Host Configuration
-Start a shell and define `DESTDIR`:
+Before creating any jail on the host system it is necessary to perform certain configurations and obtain some information from the host system.
-[source,shell]
-....
-# sh
-# export DESTDIR=/here/is/the/jail
-....
+It will be necessary to configure the man:jail[8] utility, create the necessary directories to configure and install the jails, obtain information from the host's network and check if the host uses OpenZFS or UFS as file system.
-Mount the install media as covered in man:mdconfig[8] when using the install ISO:
+[TIP]
+====
+The FreeBSD version running in the jail can not be newer that the version running in the host
+====
-[source,shell]
-....
-# mount -t cd9660 /dev/`mdconfig -f cdimage.iso` /mnt
-# cd /mnt/usr/freebsd-dist/
-....
+[[host-configuration-jail-Utility]]
+=== Jail Utility
-Extract the binaries from the tarballs on the install media into the declared destination.
-Minimally, only the base set needs to be extracted, but a complete install can be performed when preferred.
+The man:jail[8] utility manages jails.
-To install just the base system:
+To start the jails when the system boots, run the following commands:
[source,shell]
....
-# tar -xf base.txz -C $DESTDIR
+# sysrc jail_enable="YES"
+# sysrc jail_parallel_start="YES"
....
-To install everything except the kernel:
+[TIP]
+====
+With `jail_parallel_start` all configured jails will be started in the background.
+====
-[source,shell]
-....
-# for set in base ports; do tar -xf $set.txz -C $DESTDIR ; done
-....
+[[jails-networking]]
+=== Networking
-[[jails-install-source]]
-==== To build and install a Jail from source
+Networking for FreeBSD jails can be configured several different ways:
-The man:jail[8] manual page explains the procedure for building a jail:
+Host Networking Mode (IP Sharing)::
+In host networking mode, a jail shares the same networking stack as the host system.
+When a jail is created in host networking mode it uses the same network interface and IP address.
+This means that the jail doesn't have a separate IP address, and its network traffic is associated with the host's IP.
-[source,shell]
-....
-# setenv D /here/is/the/jail
-# mkdir -p $D <.>
-# cd /usr/src
-# make buildworld <.>
-# make installworld DESTDIR=$D <.>
-# make distribution DESTDIR=$D <.>
-# mount -t devfs devfs $D/dev <.>
-....
+Virtual Networks (VNET)::
+Virtual Networks are a feature of FreeBSD jails that offer more advanced and flexible networking solutions than a basic networking mode like host networking.
+VNET allows the creation of isolated network stacks for each jail, providing them with their own separate IP addresses, routing tables, and network interfaces.
+This offers a higher level of network isolation and allows jails to function as if they are running on separate virtual machines.
-<.> Selecting a location for a jail is the best starting point. This is where the jail will physically reside within the file system of the jail's host. A good choice can be [.filename]#/usr/jail/jailname#, where _jailname_ is the hostname identifying the jail. Usually, [.filename]#/usr/# has enough space for the jail file system, which for "complete" jails is, essentially, a replication of every file present in a default installation of the FreeBSD base system.
+The netgraph system::
+man:netgraph[4] is a versatile kernel framework for creating custom network configurations.
+It can be used to define how network traffic flows between jails and the host system and between different jails.
-<.> If you have already rebuilt your userland using `make world` or `make buildworld`, you can skip this step and install your existing userland into the new jail.
+[[host-configuration-directories]]
+=== Setting up the Jail Directory Tree
-<.> This command will populate the directory subtree chosen as jail's physical location on the file system with the necessary binaries, libraries, manual pages and so on.
-<.> The `distribution` target for make installs every needed configuration file. In simple words, it installs every installable file of [.filename]#/usr/src/etc/# to the [.filename]#/etc# directory of the jail environment: [.filename]#$D/etc/#.
+There is no specific place to put the files for the jails.
-<.> Mounting the man:devfs[8] file system inside a jail is not required. On the other hand, any, or almost any application requires access to at least one device, depending on the purpose of the given application. It is very important to control access to devices from inside a jail, as improper settings could permit an attacker to do nasty things in the jail. Control over man:devfs[8] is managed through rulesets which are described in the man:devfs[8] and man:devfs.conf[5] manual pages.
+Some administrators use [.filename]#/jail#, others [.filename]#/usr/jail#, and still others [.filename]#/usr/local/jails#.
+In this chapter [.filename]#/usr/local/jails# will be used.
-=== Configuring the Host
+Apart from [.filename]#/usr/local/jails# other directories will be created:
-Once a jail is installed, it can be started by using the man:jail[8] utility.
-The man:jail[8] utility takes four mandatory arguments which are described in the <<jails-synopsis>>.
-Other arguments may be specified too, e.g., to run the jailed process with the credentials of a specific user.
-The `_command_` argument depends on the type of the jail; for a _virtual system_, [.filename]#/etc/rc# is a good choice, since it will replicate the startup sequence of a real FreeBSD system.
-For a _service_ jail, it depends on the service or application that will run within the jail.
+* [.filename]#media# will contain the compressed files of the downloaded userlands.
+* [.filename]#templates# will contain the templates when using Thin Jails.
+* [.filename]#containers# will contain the jails.
-Jails are often started at boot time and the FreeBSD [.filename]#rc# mechanism provides an easy way to do this.
+When using OpenZFS, execute the following commands to create the dataset:
-[.procedure]
-* Configure jail parameters in [.filename]#jail.conf#:
-+
-[.programlisting]
+[source,shell]
....
-www {
- host.hostname = www.example.org; # Hostname
- ip4.addr = 192.168.0.10; # IP address of the jail
- path = "/usr/jail/www"; # Path to the jail
- mount.devfs; # Mount devfs inside the jail
- exec.start = "/bin/sh /etc/rc"; # Start command
- exec.stop = "/bin/sh /etc/rc.shutdown"; # Stop command
-}
+# zfs create -o mountpoint=/usr/local/jails zroot/jails
+# zfs create zroot/jails/media
+# zfs create zroot/jails/templates
+# zfs create zroot/jails/containers
....
-+
-Configure jails to start at boot time in [.filename]#rc.conf#:
-+
-[.programlisting]
-....
-jail_enable="YES" # Set to NO to disable starting of any jails
-....
-+
-The default startup of jails configured in man:jail.conf[5], will run the [.filename]#/etc/rc# script of the jail, which assumes the jail is a complete virtual system.
-For service jails, the default startup command of the jail should be changed, by setting the `exec.start` option appropriately.
-+
-[NOTE]
+[TIP]
====
-For a full list of available options, please see the man:jail.conf[5] manual page.
+In this case, `zroot` was used for the parent dataset, but other datasets could have been used.
====
-man:service[8] can be used to start or stop a jail by hand, if an entry for it exists in [.filename]#jail.conf#:
+When using UFS, execute the following commands to create the directories:
[source,shell]
....
-# service jail start www
-# service jail stop www
+# mkdir /usr/local/jails/
+# mkdir /usr/local/jails/media
+# mkdir /usr/local/jails/templates
+# mkdir /usr/local/jails/containers
....
-Jails can be shut down with man:jexec[8].
-Use man:jls[8] to identify the jail's `JID`, then use man:jexec[8] to run the shutdown script in that jail.
+[[jail-configuration-files]]
+=== Jail Configuration Files
-[source,shell]
+There are two ways to configure the jails.
+
+The first one is to add an entry for each jail to the file [.filename]#/etc/jail.conf#.
+The other option is to create a file for each jail in the directory [.filename]#/etc/jail.conf.d/#.
+
+There is no right or wrong option.
+Each administrator must choose the one that best suits their needs.
+
+In case a host system has few jails, an entry for each jail can be added in the file [.filename]#/etc/jail.conf#.
+If the host system has many jails, it is good idea to have one configuration file for each jail in the [.filename]#/etc/jail.conf.d/# directory.
+
+A typical jail entry would look like this:
+
+[.programlisting]
....
-# jls
- JID IP Address Hostname Path
- 3 192.168.0.10 www /usr/jail/www
-# jexec 3 /etc/rc.shutdown
+jailname { <.>
+ # STARTUP/LOGGING
+ exec.start = "/bin/sh /etc/rc"; <.>
+ exec.stop = "/bin/sh /etc/rc.shutdown"; <.>
+ exec.consolelog = "/var/log/jail_console_${name}.log"; <.>
+
+ # PERMISSIONS
+ allow.raw_sockets; <.>
+ exec.clean; <.>
+ mount.devfs; <.>
+
+ # HOSTNAME/PATH
+ host.hostname = "${name}"; <.>
+ path = "/usr/local/jails/containers/${name}"; <.>
+
+ # NETWORK
+ ip4.addr = 192.168.1.151; <.>
+ ip6.addr = ::ffff:c0a8:197 <.>
+ interface = em0; <.>
+}
....
-More information about this can be found in the man:jail[8] manual page.
+<.> `jailname` - Name of the jail.
+<.> `exec.start` - Command(s) to run in the jail environment when a jail is created. A typical command to run is "/bin/sh /etc/rc".
+<.> `exec.stop` - Command(s) to run in the jail environment before a jail is removed. A typical command to run is "/bin/sh /etc/rc.shutdown".
+<.> `exec.consolelog` - A file to direct command output (stdout and stderr) to.
+<.> `allow.raw_sockets` - Allow to create raw sockets. Setting this parameter allows utilities like man:ping[8] and man:traceroute[8] to operate inside the jail.
+<.> `exec.clean` - Run commands in a clean environment.
+<.> `mount.devfs` - Mount a man:devfs[5] filesystem on the chrooted [.filename]#/dev# directory, and apply the ruleset in the devfs_ruleset parameter to restrict the devices visible inside the jail.
+<.> `host.hostname` - The hostname of the jail.
+<.> `path` - The directory which is to be the root of the jail. Any commands run inside the jail, either by jail or from man:jexec[8], are run from this directory.
+<.> `ip4.addr` - IPv4 address. There are two configuration possibilities for IPv4, the first is to establish an IP or a list of IPs as has been done in the example. The other is to use `ip4` instead and set the `inherit` value to inherit the host's IP address.
+<.> `ip6.addr` - IPv6 address. There are two configuration possibilities for IPv6, the first is to establish an IP or a list of IPs as has been done in the example. The other is to use `ip6` instead and set the `inherit` value to inherit the host's IP address.
+<.> `interface` - A network interface to add the jail's IP addresses. Usually the host interface.
-[[jails-tuning]]
-== Fine Tuning and Administration
+More information about configuration variables can be found in man:jail[8] and man:jail.conf[5].
-There are several options which can be set for any jail, and various ways of combining a host FreeBSD system with jails, to produce higher level applications.
-This section presents:
+[[classic-jail]]
+== Classic Jail (Thick Jail)
-* Some of the options available for tuning the behavior and security restrictions implemented by a jail installation.
-* Some of the high-level applications for jail management, which are available through the FreeBSD Ports Collection, and can be used to implement overall jail-based solutions.
+These jails resemble a real FreeBSD system.
+They can be managed more or less like a normal host system and updated independently.
-[[jails-tuning-utilities]]
-=== System Tools for Jail Tuning in FreeBSD
+[[creating-classic-jail]]
+=== Creating a Classic Jail
-Fine tuning of a jail's configuration is mostly done by setting man:sysctl[8] variables.
-A special subtree of sysctl exists as a basis for organizing all the relevant options: the `security.jail.*` hierarchy of FreeBSD kernel options.
-Here is a list of the main jail-related sysctls, complete with their default value.
-Names should be self-explanatory, but for more information about them, please refer to the man:jail[8] and man:sysctl[8] manual pages.
+In principle, a jail only needs a hostname, a root directory, an IP address and a userland.
-* `security.jail.set_hostname_allowed: 1`
-* `security.jail.socket_unixiproute_only: 1`
-* `security.jail.sysvipc_allowed: 0`
-* `security.jail.enforce_statfs: 2`
-* `security.jail.allow_raw_sockets: 0`
-* `security.jail.chflags_allowed: 0`
-* `security.jail.jailed: 0`
+The userland for the jail can be obtained from the official FreeBSD download servers.
-These variables can be used by the system administrator of the _host system_ to add or remove some of the limitations imposed by default on the `root` user.
-Note that there are some limitations which cannot be removed.
-The `root` user is not allowed to mount or unmount file systems from within a man:jail[8].
-The `root` inside a jail may not load or unload man:devfs[8] rulesets, set firewall rules, or do many other administrative tasks which require modifications of in-kernel data, such as setting the `securelevel` of the kernel.
+Execute the following command to download the userland:
-The base system of FreeBSD contains a basic set of tools for viewing information about the active jails, and attaching to a jail to run administrative commands.
-The man:jls[8] and man:jexec[8] commands are part of the base FreeBSD system, and can be used to perform the following simple tasks:
-
-* Print a list of active jails and their corresponding jail identifier (JID), IP address, hostname and path.
-* Attach to a running jail, from its host system, and run a command inside the jail or perform administrative tasks inside the jail itself. This is especially useful when the `root` user wants to cleanly shut down a jail. The man:jexec[8] utility can also be used to start a shell in a jail to do administration in it; for example:
-+
[source,shell]
....
-# jexec 1 tcsh
+# fetch https://download.freebsd.org/ftp/releases/amd64/amd64/13.2-RELEASE/base.txz -o /usr/local/jails/media/13.2-RELEASE-base.txz
....
-[[jails-tuning-admintools]]
-=== High-Level Administrative Tools in the FreeBSD Ports Collection
-
-Among the many third-party utilities for jail administration, one of the most complete and useful is package:sysutils/ezjail[].
-It is a set of scripts that contribute to man:jail[8] management.
-Please refer to <<jails-ezjail,the handbook section on ezjail>> for more information.
+Once the download is complete it will be necessary to extract the contents into the jail directory.
-[[jails-updating]]
-=== Keeping Jails Patched and up to Date
-
-Jails should be kept up to date from the host operating system as attempting to patch userland from within the jail may likely fail as the default behavior in FreeBSD is to disallow the use of man:chflags[1] in a jail which prevents the replacement of some files.
-It is possible to change this behavior but it is recommended to use man:freebsd-update[8] to maintain jails instead.
-Use `-b` to specify the path of the jail to be updated.
-
-To update the jail to the latest patch release of the version of FreeBSD it is already running, then execute the following commands on the host:
+Execute the following commands to extract the userland into jail's directory:
[source,shell]
....
-# freebsd-update -b /here/is/the/jail fetch
-# freebsd-update -b /here/is/the/jail install
+# mkdir -p /usr/local/jails/containers/classic
+# tar -xf /usr/local/jails/media/13.2-RELEASE-base.txz -C /usr/local/jails/containers/classic --unlink
....
-To upgrade the jail to a new major or minor version, first upgrade the host system as described in crossref:cutting-edge[freebsdupdate-upgrade,“Performing Major and Minor Version Upgrades”].
-Once the host has been upgraded and rebooted, the jail can then be upgraded.
-For example to upgrade from 12.2-RELEASE to 12.3-RELEASE, on the host run:
+With the userland extracted in the jail directory, will be necessary to copy the timezone and the DNS servers files:
[source,shell]
....
-# freebsd-update -b /here/is/the/jail --currently-running 12.2-RELEASE -r 12.3-RELEASE upgrade
-# freebsd-update -b /here/is/the/jail install
-# service jail restart myjail
-# freebsd-update -b /here/is/the/jail install
+# cp /etc/resolv.conf /usr/local/jails/containers/classic/etc/resolv.conf
+# cp /etc/localtime /usr/local/jails/containers/classic/etc/localtime
....
-Then, if it was a major version upgrade, reinstall all installed packages and restart the jail again.
-This is required because the ABI version changes when upgrading between major versions of FreeBSD.
-From the host:
+With the files moved the next thing to do is update to the latest patch level executing the following command:
[source,shell]
....
-# pkg -j myjail upgrade -f
-# service jail restart myjail
+# freebsd-update -b /usr/local/jails/containers/classic/ fetch install
....
-[[jails-application]]
-== Updating Multiple Jails
+The last step will be to configure it.
+It will be necessary to add an entry to the configuration file [.filename]#/etc/jail.conf# or in [.filename]#jail.conf.d# with the data of the jail.
-The management of multiple jails can become problematic because every jail has to be rebuilt from scratch whenever it is upgraded.
-This can be time consuming and tedious if a lot of jails are created and manually updated.
+An example would be the following:
-This section demonstrates one method to resolve this issue by safely sharing as much as is possible between jails using read-only man:mount_nullfs[8] mounts, so that updating is simpler.
-This makes it more attractive to put single services, such as HTTP, DNS, and SMTP, into individual jails.
-Additionally, it provides a simple way to add, remove, and upgrade jails.
+[.programlisting]
+....
+classic {
+ # STARTUP/LOGGING
+ exec.start = "/bin/sh /etc/rc";
+ exec.stop = "/bin/sh /etc/rc.shutdown";
+ exec.consolelog = "/var/log/jail_console_${name}.log";
-[NOTE]
-====
-Simpler solutions exist, such as ezjail, which provides an easier method of administering FreeBSD jails but is less versatile than this setup.
-ezjail is covered in more detail in <<jails-ezjail>>.
-====
+ # PERMISSIONS
+ allow.raw_sockets;
+ exec.clean;
+ mount.devfs;
+
+ # HOSTNAME/PATH
+ host.hostname = "${name}";
+ path = "/usr/local/jails/containers/${name}";
+
+ # NETWORK
+ ip4.addr = 192.168.1.151;
+ interface = em0;
+}
+....
-The goals of the setup described in this section are:
+And then execute the following command to start the jail:
-* Create a simple and easy to understand jail structure that does not require running a full installworld on each and every jail.
-* Make it easy to add new jails or remove existing ones.
-* Make it easy to update or upgrade existing jails.
-* Make it possible to run a customized FreeBSD branch.
-* Be paranoid about security, reducing as much as possible the possibility of compromise.
-* Save space and inodes, as much as possible.
+[source,shell]
+....
+# service jail start classic
+....
-This design relies on a single, read-only master template which is mounted into each jail and one read-write device per jail.
-A device can be a separate physical disc, a partition, or a vnode backed memory device.
-This example uses read-write nullfs mounts.
+More information on how to manage jails can be found in the section <<jail-management>>.
-The file system layout is as follows:
+[[thin-jail]]
+== Thin Jails
-* The jails are based under the [.filename]#/home# partition.
-* Each jail will be mounted under the [.filename]#/home/j# directory.
-* The template for each jail and the read-only partition for all of the jails is [.filename]#/home/j/mroot#.
-* A blank directory will be created for each jail under the [.filename]#/home/j# directory.
-* Each jail will have a [.filename]#/s# directory that will be linked to the read-write portion of the system.
-* Each jail will have its own read-write system that is based upon [.filename]#/home/j/skel#.
-* The read-write portion of each jail will be created in [.filename]#/home/js#.
+Although the Thin Jails use the same technology, the creation is different.
+Thin jails can be created using OpenZFS snapshots or using templates and NullFS.
+The use of OpenZFS snapshots and templates using NullFS have certain advantages over classic jails,
+such as being able to create them faster from snapshots or being able to update multiple jails using NullFS.
-[[jails-service-jails-template]]
-=== Creating the Template
+[[creating-thin-jail-openzfs-snapshots]]
+=== Creating a Thin Jail using OpenZFS Snapshots
-This section describes the steps needed to create the master template.
+Due to the good integration between FreeBSD and OpenZFS it is very easy to create new Thin Jails using OpenZFS Snapshots.
-It is recommended to first update the host FreeBSD system to the latest -RELEASE branch using the instructions in crossref:cutting-edge[makeworld,“Updating FreeBSD from Source”].
-Additionally, this template uses the package:sysutils/cpdup[] package or port and link:{handbook}mirrors/#git[Git] will be used to download the FreeBSD Ports Collection.
+To create a Thin Jail using OpenZFS Snapshots the first step will be to create a template.
+
+The templates will only be used to create new jails, for this reason they are created in "read-only" mode, in order to create jails easily with an immutable base.
+
+To create the dataset to save the template execute the following command:
-[.procedure]
-. First, create a directory structure for the read-only file system which will contain the FreeBSD binaries for the jails. Then, change directory to the FreeBSD source tree and install the read-only file system to the jail template:
-+
[source,shell]
....
-# mkdir /home/j /home/j/mroot
-# cd /usr/src
-# make installworld DESTDIR=/home/j/mroot
+# zfs create -p zroot/jails/templates/13.2-RELEASE
....
-. Next, prepare a FreeBSD Ports Collection for the jails as well as a FreeBSD source tree, which is required for mergemaster:
-+
+Then execute the following command to download the userland:
+
[source,shell]
....
-# cd /home/j/mroot
-# mkdir usr/ports
-# git clone -o freebsd https://git.FreeBSD.org/ports.git /home/j/mroot/usr/ports
-# cpdup /usr/src /home/j/mroot/usr/src
+# fetch https://download.freebsd.org/ftp/releases/amd64/amd64/13.2-RELEASE/base.txz -o /usr/local/jails/media/13.2-RELEASE-base.txz
....
-. Create a skeleton for the read-write portion of the system:
-+
+Once the download is complete it will be necessary to extract the contents in the template directory executing the following command:
+
[source,shell]
....
-# mkdir /home/j/skel /home/j/skel/home /home/j/skel/usr-X11R6 /home/j/skel/distfiles
-# mv etc /home/j/skel
-# mv usr/local /home/j/skel/usr-local
-# mv tmp /home/j/skel
-# mv var /home/j/skel
-# mv root /home/j/skel
+# tar -xf /usr/local/jails/media/13.2-RELEASE-base.txz -C /usr/local/jails/templates/13.2-RELEASE --unlink
....
-. Use mergemaster to install missing configuration files. Then, remove the extra directories that mergemaster creates:
-+
+With the userland extracted in the templates directory, it will be necessary to copy the timezone and the DNS servers files to the template directory executing the following command:
+
[source,shell]
....
-# mergemaster -t /home/j/skel/var/tmp/temproot -D /home/j/skel -i
-# cd /home/j/skel
-# rm -R bin boot lib libexec mnt proc rescue sbin sys usr dev
+# cp /etc/resolv.conf /usr/local/jails/templates/13.2-RELEASE/etc/resolv.conf
+# cp /etc/localtime /usr/local/jails/templates/13.2-RELEASE/etc/localtime
....
-. Now, symlink the read-write file system to the read-only file system. Ensure that the symlinks are created in the correct [.filename]#s/# locations as the creation of directories in the wrong locations will cause the installation to fail.
-+
+The next thing to do is update to the latest patch level executing the following command:
+
[source,shell]
....
-# cd /home/j/mroot
-# mkdir s
-# ln -s s/etc etc
-# ln -s s/home home
-# ln -s s/root root
-# ln -s ../s/usr-local usr/local
-# ln -s ../s/usr-X11R6 usr/X11R6
-# ln -s ../../s/distfiles usr/ports/distfiles
-# ln -s s/tmp tmp
-# ln -s s/var var
+# freebsd-update -b /usr/local/jails/templates/13.2-RELEASE/ fetch install
....
-. As a last step, create a generic [.filename]#/home/j/skel/etc/make.conf# containing this line:
-+
-[.programlisting]
+Once the update is finished the template will be ready.
+
+To create the OpenZFS Snapshot from the template execute the following command:
+
+[source,shell]
....
-WRKDIRPREFIX?= /s/portbuild
+# zfs snapshot zroot/jails/templates/13.2-RELEASE@base
....
-+
-This makes it possible to compile FreeBSD ports inside each jail.
-Remember that the ports directory is part of the read-only system.
-The custom path for `WRKDIRPREFIX` allows builds to be done in the read-write portion of every jail.
-[[jails-service-jails-creating]]
-=== Creating Jails
+Once the OpenZFS Snapshot has been created infinite jails can be created using the OpenZFS clone function.
-The jail template can now be used to setup and configure the jails in [.filename]#/etc/rc.conf#.
-This example demonstrates the creation of 3 jails: `NS`, `MAIL` and `WWW`.
+To create a Thin Jail named `thinjail` execute the following command:
-[.procedure]
-. Add the following lines to [.filename]#/etc/fstab#, so that the read-only template for the jails and the read-write space will be available in the respective jails:
-+
-[.programlisting]
-....
-/home/j/mroot /home/j/ns nullfs ro 0 0
-/home/j/mroot /home/j/mail nullfs ro 0 0
-/home/j/mroot /home/j/www nullfs ro 0 0
-/home/js/ns /home/j/ns/s nullfs rw 0 0
-/home/js/mail /home/j/mail/s nullfs rw 0 0
-/home/js/www /home/j/www/s nullfs rw 0 0
-....
-+
-To prevent fsck from checking nullfs mounts during boot and dump from backing up the read-only nullfs mounts of the jails, the last two columns are both set to `0`.
-. Configure the jails in [.filename]#/etc/rc.conf#:
-+
-[.programlisting]
-....
-jail_enable="YES"
-jail_set_hostname_allow="NO"
-jail_list="ns mail www"
-jail_ns_hostname="ns.example.org"
-jail_ns_ip="192.168.3.17"
-jail_ns_rootdir="/usr/home/j/ns"
-jail_ns_devfs_enable="YES"
-jail_mail_hostname="mail.example.org"
-jail_mail_ip="192.168.3.18"
-jail_mail_rootdir="/usr/home/j/mail"
-jail_mail_devfs_enable="YES"
-jail_www_hostname="www.example.org"
-jail_www_ip="62.123.43.14"
-jail_www_rootdir="/usr/home/j/www"
-jail_www_devfs_enable="YES"
-....
-+
-The `jail__name__rootdir` variable is set to [.filename]#/usr/home# instead of [.filename]#/home# because the physical path of [.filename]#/home# on a default FreeBSD installation is [.filename]#/usr/home#.
-The `jail__name__rootdir` variable must _not_ be set to a path which includes a symbolic link, otherwise the jails will refuse to start.
-. Create the required mount points for the read-only file system of each jail:
-+
[source,shell]
....
-# mkdir /home/j/ns /home/j/mail /home/j/www
+# zfs clone zroot/jails/templates/13.2-RELEASE@base zroot/jails/containers/thinjail
....
-. Install the read-write template into each jail using package:sysutils/cpdup[]:
-+
-[source,shell]
+The last step is to configure it.
+It will be necessary to add an entry to the configuration file [.filename]#/etc/jail.conf# or in [.filename]#jail.conf.d# with the data of the jail.
+
+An example would be the following:
+
+[.programlisting]
....
-# mkdir /home/js
-# cpdup /home/j/skel /home/js/ns
-# cpdup /home/j/skel /home/js/mail
-# cpdup /home/j/skel /home/js/www
+thinjail {
+ # STARTUP/LOGGING
+ exec.start = "/bin/sh /etc/rc";
+ exec.stop = "/bin/sh /etc/rc.shutdown";
+ exec.consolelog = "/var/log/jail_console_${name}.log";
+
+ # PERMISSIONS
+ allow.raw_sockets;
+ exec.clean;
+ mount.devfs;
+
+ # HOSTNAME/PATH
+ host.hostname = "${name}";
+ path = "/usr/local/jails/containers/${name}";
+
+ # NETWORK
+ ip4 = inherit;
+ interface = em0;
+}
....
-. In this phase, the jails are built and prepared to run. First, mount the required file systems for each jail, and then start them:
-+
+And then execute the following command to start the jail:
+
[source,shell]
....
-# mount -a
-# service jail start
+# service jail start thinjail
....
-The jails should be running now.
-To check if they have started correctly, use `jls`.
-Its output should be similar to the following:
+More information on how to manage jails can be found in the section <<jail-management>>.
+
+[[creating-thin-jail-nullfs]]
+=== Creating a Thin Jail using NullFS
+
+Combining Thin Jails and NullFS, a jail can be created with reduced duplication of system files by using Thin Jail techniques,
+and use NullFS to selectively share specific directories from the host system into the jail.
+
+The first step is to create the dataset to save the template, execute the following command if using OpenZFS:
[source,shell]
....
-# jls
- JID IP Address Hostname Path
- 3 192.168.3.17 ns.example.org /home/j/ns
- 2 192.168.3.18 mail.example.org /home/j/mail
- 1 62.123.43.14 www.example.org /home/j/www
+# zfs create -p zroot/jails/templates/13.2-RELEASE-base
....
-At this point, it should be possible to log onto each jail, add new users, or configure daemons.
-The `JID` column indicates the jail identification number of each running jail.
-Use the following command to perform administrative tasks in the jail whose JID is `3`:
+Or this one if using UFS:
[source,shell]
....
-# jexec 3 tcsh
+# mkdir /usr/local/jails/templates/13.2-RELEASE-base
....
-[[jails-service-jails-upgrading]]
-=== Upgrading
+Then execute the following command to download the userland:
-The design of this setup provides an easy way to upgrade existing jails while minimizing their downtime.
-Also, it provides a way to roll back to the older version should a problem occur.
-
-[.procedure]
-. The first step is to upgrade the host system. Then, create a new temporary read-only template in [.filename]#/home/j/mroot2#.
-+
[source,shell]
....
-# mkdir /home/j/mroot2
-# cd /usr/src
-# make installworld DESTDIR=/home/j/mroot2
-# cd /home/j/mroot2
-# cpdup /usr/src usr/src
-# mkdir s
+# fetch https://download.freebsd.org/ftp/releases/amd64/amd64/13.2-RELEASE/base.txz -o /usr/local/jails/media/13.2-RELEASE-base.txz
....
-+
-The `installworld` creates a few unnecessary directories, which should be removed:
-+
+
+Once the download is complete it will be necessary to extract the contents in the template directory executing the following command:
+
[source,shell]
....
-# chflags -R 0 var
-# rm -R etc var root usr/local tmp
+# tar -xf /usr/local/jails/media/13.2-RELEASE-base.txz -C /usr/local/jails/templates/13.2-RELEASE-base --unlink
....
-. Recreate the read-write symlinks for the master file system:
-+
+Once the userland is extracted n the templates directory, it will be necessary to copy the timezone and the DNS servers files to the template directory executing the following command:
+
[source,shell]
....
-# ln -s s/etc etc
-# ln -s s/root root
-# ln -s s/home home
-# ln -s ../s/usr-local usr/local
-# ln -s ../s/usr-X11R6 usr/X11R6
-# ln -s s/tmp tmp
-# ln -s s/var var
+# cp /etc/resolv.conf /usr/local/jails/templates/13.2-RELEASE-base/etc/resolv.conf
+# cp /etc/localtime /usr/local/jails/templates/13.2-RELEASE-base/etc/localtime
....
-. Next, stop the jails:
-+
+With the files moved to the template, the next thing to do is update to the latest patch level executing the following command:
+
[source,shell]
....
-# service jail stop
+# freebsd-update -b /usr/local/jails/templates/13.2-RELEASE-base/ fetch install
....
-. Unmount the original file systems as the read-write systems are attached to the read-only system ([.filename]#/s#):
-+
+In addition to the base template, it is also necessary to create a directory where the `skeleton` will be located.
+Some directories will be copied from the template to the `skeleton`.
+
+Execute the following command to create the dataset for the `skeleton` in case of using OpenZFS:
+
[source,shell]
....
-# umount /home/j/ns/s
-# umount /home/j/ns
-# umount /home/j/mail/s
-# umount /home/j/mail
-# umount /home/j/www/s
-# umount /home/j/www
+# zfs create -p zroot/jails/templates/13.2-RELEASE-skeleton
....
-. Move the old read-only file system and replace it with the new one. This will serve as a backup and archive of the old read-only file system should something go wrong. The naming convention used here corresponds to when a new read-only file system has been created. Move the original FreeBSD Ports Collection over to the new file system to save some space and inodes:
-+
+Or this one in case of using UFS:
+
[source,shell]
....
-# cd /home/j
-# mv mroot mroot.20060601
-# mv mroot2 mroot
-# mv mroot.20060601/usr/ports mroot/usr
+# mkdir /usr/local/jails/templates/13.2-RELEASE-skeleton
....
-. At this point the new read-only template is ready, so the only remaining task is to remount the file systems and start the jails:
-+
+Then create the `skeleton` directories.
+The `skeleton` directories will hold the local directories of the jails.
+
+Execute the following commands to create the directories:
*** 976 LINES SKIPPED ***