docs/62834: Diskless operation handbook section update
Jean-Francois Dockes
jean-francois.dockes at wanadoo.fr
Wed Mar 3 12:30:27 UTC 2004
The following reply was made to PR docs/62834; it has been noted by GNATS.
From: Jean-Francois Dockes <jean-francois.dockes at wanadoo.fr>
To: freebsd-gnats-submit at FreeBSD.org
Cc:
Subject: Re: docs/62834: Diskless operation handbook section update
Date: Wed, 3 Mar 2004 13:28:57 +0100
I have integrated initial remarks to the handbook advanced networking
diskless section update.
The revised text can be found at:
http://perso.wanadoo.fr/dockes/diskless/network-diskless.html
The unified diffs follow, or can be found at:
http://perso.wanadoo.fr/dockes/diskless/udiffs.txt
I really believe that the new version is better than the previous one
(which I wrote). It corrects some errors and contains updated material for
FreeBSD 5.x
I would appreciate it if someone could find the time to review and possibly
commit the new text.
Regards,
J.F. Dockes
--- chapter.sgml.old Wed Mar 3 12:48:33 2004
+++ chapter.sgml Wed Mar 3 12:58:57 2004
@@ -2402,10 +2402,10 @@
<indexterm><primary>diskless operation</primary></indexterm>
<para>A FreeBSD machine can boot over the network and operate without a
- local disk, using filesystems mounted from an NFS server. No system
- modification is necessary, beyond standard configuration files.
- Such a system is easy to set up because all the necessary elements
- are readily available:</para>
+ local disk, using filesystems mounted from an NFS server. No
+ system modification is necessary, beyond standard configuration
+ files. Such a system is relatively easy to set up because all
+ the necessary elements are readily available:</para>
<itemizedlist>
<listitem>
<para>There are at least two possible methods to load the kernel over
@@ -2451,10 +2451,10 @@
<para>There are many ways to set up diskless workstations. Many
elements are involved, and most can be customized to suit local
- taste. The following will describe the setup of a complete system,
- emphasizing simplicity and compatibility with the
- standard FreeBSD startup scripts. The system described has the
- following characteristics:</para>
+ taste. The following will describe variations on the setup of a
+ complete system, emphasizing simplicity and compatibility with
+ the standard FreeBSD startup scripts. The system described has
+ the following characteristics:</para>
<itemizedlist>
<listitem>
@@ -2466,12 +2466,14 @@
configuration files overridden by ones specific to diskless
operation or, possibly, to the workstation they belong to.</para>
<para>The parts of the <filename>root</filename> which have to be
- writable are overlaid with &man.mfs.8; filesystems. Any changes
- will be lost when the system reboots.</para>
+ writable are overlaid with &man.mfs.8; (FreeBSD 4.X) or
+ &man.md.4; (FreeBSD 5.X) filesystems. Any changes will be lost
+ when the system reboots.</para>
</listitem>
<listitem>
- <para>The kernel is loaded by <application>etherboot
- </application>, using DHCP (or BOOTP) and TFTP.</para>
+ <para>The kernel is transferred and loaded either with
+ <application>etherboot</application>, or <acronym>PXE</acronym>
+ as some situations may mandate the use of either method.</para>
</listitem>
</itemizedlist>
@@ -2480,6 +2482,111 @@
other hosts.</para>
</caution>
+ <para>All the information in this section has been tested
+ using FreeBSD releases 4.9-RELEASE-p2 and 5.2-RELEASE. The text is
+ primarily structured for 4.X usage. Notes have been inserted where
+ appropriate to indicate 5.X changes.</para>
+
+
+
+ <sect2>
+ <title>Background information</title>
+
+ <para>Setting up diskless workstations is both relatively
+ straightforward and prone to errors. These are sometimes
+ difficult to diagnose for a number of reasons. For example:
+ <itemizedlist>
+ <listitem>
+ <para>Compile time options may determine the possible values
+ or syntax of runtime parameters.</para>
+ </listitem>
+ <listitem>
+ <para>Error messages are often cryptic or totally
+ absent.</para>
+ </listitem>
+ </itemizedlist>
+
+ <para>In this context, having some knowledge of the background
+ mechanisms involved is very useful to solve the problems that
+ may arise.</para>
+
+ <para>Several operations need to be performed for a successful
+ bootstrap:</para>
+
+ <itemizedlist>
+ <listitem>
+ <para>The machine needs to obtain initial
+ parameters such as its IP address, executable file name,
+ server name, root path. This is done using the DHCP or
+ BOOTP protocols. DHCP is a compatible extension of BOOTP,
+ and uses the same port numbers and basic packet
+ format.</para>
+ <para>It is possible to configure a system to use only BOOTP.
+ The &man.bootpd.8; server program is included in the
+ base FreeBSD system.</para>
+
+ <para>However, DHCP has a number of advantages over BOOTP (nicer
+ configuration files, possibility of using PXE, plus many
+ others not directly related to diskless operation), and
+ we shall describe mainly a DHCP configuration, with
+ equivalent exemples using &man.bootpd.8; when possible.
+ The sample configuration will use the ISC DHCP software
+ package (release 3.0.1.r11 was installed on the test
+ server: ).</para>
+ </listitem>
+
+ <listitem>
+ <para>The machine needs to transfer one or several
+ programs to local memory. Either TFTP or NFS are used. The
+ choice between TFTP and NFS is a compile time option in
+ several places. A common source of error is to specify
+ file names for the wrong protocol: TFTP typically
+ transfers all files from a single directory on the server,
+ and would expect file names relative to this
+ directory. NFS needs absolute file paths.</para>
+ </listitem>
+
+ <listitem>
+ <para>The possible intermediate bootstrap programs and the
+ kernel need to be initialized and executed. There are
+ several important variations in this area:</para>
+ <itemizedlist>
+ <listitem>
+ <para>PXE will load &man.pxeboot.8;,
+ which is a modified version of the FreeBSD third stage
+ loader. The loader will obtain most parameters necessary
+ to system startup, and leave them in the kernel
+ environment before transferring control. It is possible to
+ use a GENERIC kernel in this case.</para>
+ </listitem>
+
+ <listitem>
+ <para><application>etherboot</application>, will directly
+ load the kernel, with less preparation. A kernel with
+ specific options will need to be built.</para>
+ </listitem>
+ </itemizedlist>
+
+ <para>PXE and <application>etherboot</application> work
+ equally well with 4.X systems. Because 5.X kernels
+ normally let the loader do more work for them, PXE is
+ preferred for 5.X systems.</para>
+ <para>If your BIOS and network
+ cards support PXE, you should probably use it. However,
+ it is still possible to start a 5.2 system with
+ <application>etherboot</application></para>
+
+ </listitem>
+
+ <listitem>
+ <para>Finally, the machine needs to access its filesystems. NFS
+ is used in all cases.</para>
+ </listitem>
+ </itemizedlist>
+
+ <para>See also &man.diskless.8;</para>
+
+ </sect2>
<sect2>
<title>Setup Instructions</title>
@@ -2491,31 +2598,7 @@
<secondary>booting</secondary>
</indexterm>
- <para>There are two protocols that are commonly used to boot a
- workstation that retrieves its configuration over the network: BOOTP
- and DHCP. They are used at several points in the workstation
- bootstrap:</para>
- <itemizedlist>
- <listitem><para><application>etherboot</application> uses
- DHCP (by default) or BOOTP (needs a configuration option) to
- find the kernel. (PXE uses DHCP).</para>
- </listitem>
- <listitem><para>The kernel uses BOOTP to locate the NFS
- root.</para>
- </listitem>
- </itemizedlist>
-
- <para>It is possible to configure a system to use only BOOTP.
- The &man.bootpd.8; server program is included in the
- base FreeBSD system.</para>
-
- <para>However, DHCP has a number of advantages over BOOTP (nicer
- configuration files, possibility of using PXE, plus many others
- not directly related to diskless operation), and we shall describe
- both a pure BOOTP, and a BOOTP+DHCP configuration, with an
- emphasis on the latter, which will use the ISC DHCP software
- package.</para>
-
+
<sect4>
<title>Configuration Using ISC DHCP</title>
<indexterm>
@@ -2526,7 +2609,7 @@
<para>The <application>isc-dhcp</application> server can answer
both BOOTP and DHCP requests.</para>
- <para>As of release 4.4, <application>isc-dhcp
+ <para>As of release 4.9, <application>isc-dhcp
3.0</application> is not part of the base
system. You will first need to install the
<filename role="package">net/isc-dhcp3-server</filename> port or the
@@ -2536,7 +2619,9 @@
<para>Once <application>isc-dhcp</application> is installed, it
needs a configuration file to run, (normally named
<filename>/usr/local/etc/dhcpd.conf</filename>). Here follows
- a commented example:</para>
+ a commented example, where host <literal>margaux</literal>
+ uses <application>etherboot</application> and host
+ <literal>corbieres</literal> uses PXE.</para>
<programlisting>
default-lease-time 600;
@@ -2556,9 +2641,16 @@
hardware ethernet 01:23:45:67:89:ab;
fixed-address margaux.example.com;
next-server 192.168.4.4;<co id="co-dhcp-next-server">
- filename "/tftpboot/kernel.diskless";<co id="co-dhcp-filename">
+ filename "/data/misc/kernel.diskless";<co id="co-dhcp-filename">
option root-path "192.168.4.4:/data/misc/diskless";<co id="co-dhcp-root-path">
}
+ host corbieres {
+ hardware ethernet 00:02:b3:27:62:df;
+ fixed-address corbieres.example.com;
+ filename "pxeboot";<co id="co-dhcp-filenamePxe">
+ next-server 192.168.4.4;
+ option root-path "192.168.4.4:/data/misc/diskless";
+ }
}
</programlisting>
@@ -2573,26 +2665,37 @@
</callout>
<callout arearefs="co-dhcp-next-server"><para>The
- <literal>next-server</literal> directive designates
- the TFTP server (the default is to use the same host as the
- DHCP server).</para>
+ <literal>next-server</literal> directive designates the TFTP
+ or NFS server to use for loading loader or kernel file (the
+ default is to use the same host as the DHCP server).</para>
</callout>
- <callout arearefs="co-dhcp-filename"><para>The
- <literal>filename</literal> directive defines the file that
- <application>etherboot</application> will load as a
- kernel.
- <note><para>PXE appears to prefer a relative file
- name, and it loads <command>pxeboot</command>, not the
- kernel (<literal>option filename
- "pxeboot"</literal>).</para>
- </note>
- </para>
+ <callout arearefs="co-dhcp-filename"><para>The
+ <literal>filename</literal> directive defines the file that
+ <application>etherboot</application> or PXE will load for the
+ next execution step. It must be specified according to the
+ transfer method used. <application>etherboot</application> can
+ be compiled to use NFS or TFTP. The FreeBSD port
+ configures NFS by default.</para>
+ </callout>
+
+ <callout arearefs="co-dhcp-filenamePxe"><para>PXE uses TFTP,
+ which is why a relative file name is used here (this may
+ depend on the TFTP server configuration, but would be fairly
+ typical). Also, PXE loads <filename>pxeboot</filename>, not
+ the kernel. There are other interesting possibilities, like
+ loading <filename>pxeboot</filename> from a FreeBSD cdrom
+ <filename>/boot</filename> directory (as &man.pxeboot.8; can
+ load a GENERIC kernel, this makes it possible to use PXE to
+ boot from a remote cdrom).</para>
</callout>
<callout arearefs="co-dhcp-root-path"><para>The
- <literal>root-path</literal> option defines the path to
- the root filesystem, in usual NFS notation.</para>
+ <literal>root-path</literal> option defines the path to the
+ root filesystem, in usual NFS notation. When using PXE, it is
+ possible to leave off the host's IP as long as you do not
+ enable the kernel option BOOTP. The NFS server will then be
+ the same as the TFTP one.</para>
</callout>
</calloutlist>
@@ -2605,7 +2708,7 @@
</indexterm>
<para>Here follows an equivalent <command>bootpd</command>
- configuration. This would be found in
+ configuration (reduced to one client). This would be found in
<filename>/etc/bootptab</filename>.</para>
<para>Please note that <application>etherboot</application>
@@ -2656,6 +2759,10 @@
of everything. Else refer to <xref linkend="ports"> for
information about ports and packages.</para>
+ <para>You can change the <application>etherboot</application>
+ configuration (ie: to use TFTP instead of NFS) by editing the
+ Config file in the etherboot source directory.</para>
+
<para>For our setup, we shall use a boot floppy. For other methods
(PROM, or dos program), please refer to the
<application>etherboot</application> documentation.</para>
@@ -2677,6 +2784,30 @@
</sect3>
+ <sect3>
+ <title>Booting with PXE</title>
+
+ <para>By default, the &man.pxeboot.8; loader loads the kernel via NFS.
+ It can be compiled to use TFTP instead by specifying the
+ LOADER_TFTP_SUPPORT option in
+ <filename>/etc/make.conf</filename> . See the comments in
+ <filename>/etc/defaults/make.conf</filename> (or
+ <filename>/usr/share/examples/etc/make.conf</filename> for 5.x
+ systems) for instructions.</para>
+
+ <para>There are two other undocumented
+ <filename>make.conf</filename> options which may be useful for
+ setting up a serial console diskless machine:
+ BOOT_PXELDR_PROBE_KEYBOARD, and BOOT_PXELDR_ALWAYS_SERIAL (the
+ latter only exists on FreeBSD 5.x).</para>
+
+
+ <para>To use PXE when the machine starts, you will usually
+ need to select the <literal>Boot from network</literal> option
+ in your BIOS setup, or type a function key during the PC
+ initialization.</para>
+
+ </sect3>
<sect3>
<title>Configuring the TFTP and NFS Servers</title>
@@ -2690,8 +2821,10 @@
<secondary>diskless operation</secondary>
</indexterm>
- <para>You need to enable <command>tftpd</command> on the TFTP
- server:</para>
+ <para>If any of the boostrap programs (PXE, pxeboot, or etherboot)
+ are configured to use TFTP, you need to enable
+ <command>tftpd</command> on the file server:</para>
+
<procedure>
<step>
<para>Create a directory from which <command>tftpd</command>
@@ -2702,7 +2835,7 @@
<para>Add this line to your
<filename>/etc/inetd.conf</filename>:</para>
- <programlisting>tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot</programlisting>
+ <programlisting>tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot</programlisting>
<note><para>It appears that at least some PXE versions want
the TCP version of TFTP. In this case, add a second line,
@@ -2722,7 +2855,7 @@
location is set in both <filename>inetd.conf</filename> and
<filename>dhcpd.conf</filename>.</para>
- <para>You also need to enable NFS and export the
+ <para>In all cases, you also need to enable NFS and export the
appropriate filesystem on the NFS server.</para>
<procedure>
@@ -2738,7 +2871,7 @@
point and replace <replaceable>margaux</replaceable>
with the name of the diskless workstation):</para>
- <programlisting><replaceable>/data/misc</replaceable> -alldirs -ro <replaceable>margaux</replaceable></programlisting>
+ <programlisting><replaceable>/data/misc</replaceable> -alldirs -ro <replaceable>margaux corbieres</replaceable></programlisting>
</step>
<step>
<para>Tell <command>mountd</command> to reread its configuration
@@ -2759,23 +2892,45 @@
<secondary>kernel configuration</secondary>
</indexterm>
- <para>Create a kernel configuration file for the diskless client
- with the following options (in addition to the usual
- ones):</para>
+ <para>If using <application>etherboot</application>, you need to
+ create a kernel configuration file for the diskless client
+ with the following options (in addition to the usual ones):</para>
<programlisting>
- options BOOTP # Use BOOTP to obtain IP address/hostname
- options BOOTP_NFSROOT # NFS mount root filesystem using BOOTP info
- options BOOTP_COMPAT # Workaround for broken bootp daemons.
+ options BOOTP # Use BOOTP to obtain IP address/hostname
+ options BOOTP_NFSROOT # NFS mount root filesystem using BOOTP info
</programlisting>
- <para>You may also want to use <literal>BOOTP_NFSV3</literal> and
- <literal>BOOTP_WIRED_TO</literal> (refer to <filename>LINT</filename>).</para>
-
+ <para>You may also want to use <literal>BOOTP_NFSV3</literal>,
+ <literal>BOOT_COMPAT</literal> and
+ <literal>BOOTP_WIRED_TO</literal> (refer to
+ <filename>LINT</filename>).</para>
+
+ <para>These option names are historical and slightly
+ misleading as they actually enable indifferent use of DHCP or BOOTP
+ inside the kernel (it is also possible to force strict BOOTP or
+ DHCP use).</para>
+
<para>Build the kernel (See <xref linkend="kernelconfig">),
- and copy it to the tftp directory, under the name listed
- in <filename>dhcpd.conf</filename>.</para>
+ and copy it to the place specified in
+ <filename>dhcpd.conf</filename>.</para>
+ <note><para>When using PXE, building a kernel with the above
+ options is not strictly necessary. Enabling them will cause more
+ DHCP requests to be issued during kernel startup, with a small
+ risk of inconsistency between the new values and those retrieved
+ by pxeboot in some special cases. The advantage of using them is
+ that the host name will be set as a side effect. Else you will
+ need to set the host name by another method, for example in a
+ client-specific <filename>rc.conf</filename> file.</para> </note>
+
+ <note><para>In order to be loadable with
+ <application>etherboot</application>, a 5.x kernel needs to have
+ the device hints compiled in. You would typically set the
+ following option in the configuration file (see the NOTES
+ configuration comments file).
+ <programlisting>hints "GENERIC.hints" </programlisting>
+ </para></note>
</sect3>
@@ -2798,6 +2953,13 @@
the place where the filesystem will be created (the
<literal>DEST</literal> variable).</para>
+ <note><para>The <filename>clone_root</filename> script distributed
+ with FreeBSD 5.2 is out of date relative to the 4.x version, and
+ also needs a few specific 5.x adjustments. Please refer to PR
+ <ulink url="http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/62417">
+ misc/62417</ulink>.</para>
+ </note>
+
<para>Refer to the comments at the top of the script for
instructions. They explain how the base filesystem is built,
and how files may be selectively overridden by versions specific
@@ -2812,39 +2974,68 @@
other examples in the <filename>diskless</filename> directory,
they actually document a configuration method which is distinct
from the one used by <filename>clone_root</filename> and
- <filename>/etc/rc.diskless[12]</filename>, which is a little
- confusing. Use them for reference only, except if you prefer
- the method that they describe, in which case you will need
+ the system startup scripts in <filename>/etc</filename>, which
+ is a little confusing. Use them for reference only, except if
+ you prefer the method that they describe, in which case you will need
customized <filename>rc</filename> scripts.</para>
+
+ <note><para>When using PXE, it may happen that the loader will
+ get a wrong idea of the root filesystem to be used by the
+ kernel (this will be the case if the cloned root
+ <filename>/etc/fstab</filename> is a copy of the server's,
+ as the client-specific version will not be visible by the
+ loader, which looks for hints there).</para>
+ <para>This will cause
+ error messages at the end of the kernel initialization,
+ when the kernel tries to mount the wrong root and fails,
+ or even actual errors if it succeeds because a local disk
+ happens to be present. </para>
+
+ <para>It is possible to prevent this problem from happening
+ by inserting the following line in
+ <filename>/boot/loader.conf</filename>:</para>
+ <programlisting>
+ vfs.root.mountfrom=""
+ </programlisting>
+ </note>
+
+
</sect3>
<sect3>
<title>Configuring Swap</title>
<para>If needed, a swap file located on the server can be
- accessed via NFS. The exact <filename>bootptab</filename>
- or <filename>dhcpd.conf</filename> options are not clearly
- documented at this time. The following configuration
- suggestions have been reported to work in some installations
- using isc-dhcp 3.0rc11.</para>
+ accessed via NFS. One of the methods commonly used to do this
+ has been discontinued in release 5.X.</para>
+
+ <sect4>
+ <title>NFS swap with FreeBSD 4.x</title>
+
+ <para>The swap file location and size can be specified with
+ BOOTP/DHCP FreeBSD-specific options 128 and 129. Examples of
+ configuration files for isc-dhcp 3 or bootpd follow.</para>
+
<procedure>
<step><para>Add the following lines to
<filename>dhcpd.conf</filename>:</para>
<programlisting>
- # Global section
- option swap-path code 128 = string;
- option swap-size code 129 = integer 32;
-
- host margaux {
- ... # Standard lines, see above
- option swap-path <replaceable>"192.168.4.4:/netswapvolume/netswap"</replaceable>;
- option swap-size <replaceable>64000</replaceable>;
- }
+ # Global section
+ option swap-path code 128 = string;
+ option swap-size code 129 = integer 32;
+
+ host margaux {
+ ... # Standard lines, see above
+ option swap-path <replaceable>"192.168.4.4:/netswapvolume/netswap"</replaceable>;
+ option swap-size <replaceable>64000</replaceable>;
+ }
</programlisting>
- <para>The idea is that, at least for a FreeBSD client,
- DHCP/BOOTP option code 128 is the path to the NFS swap file,
- and option code 129 is the swap size in kilobytes. Older
- versions of <command>dhcpd</command> allowed a syntax of
+
+ <para><literal>swap-path</literal> is the path to a directory
+ where swap files will be located. Each file will be named
+ <filename>swap.<replaceable>clientIp</replaceable></filename>.</para>
+
+ <para>Older versions of <command>dhcpd</command> used a syntax of
<literal>option option-128 "...</literal>, which is no
longer supported.</para>
<para><filename>/etc/bootptab</filename> would use the
@@ -2860,12 +3051,12 @@
<step>
<para>On the NFS swap file server, create the swap
file(s)</para>
- <screen>
- &prompt.root; <userinput>mkdir <replaceable>/netswapvolume/netswap</replaceable></userinput>
- &prompt.root; <userinput>cd <replaceable>/netswapvolume/netswap</replaceable></userinput>
- &prompt.root; <userinput>dd if=/dev/zero bs=1024 count=<replaceable>64000</replaceable> of=swap.<replaceable>192.168.4.6</replaceable></userinput>
- &prompt.root; <userinput>chmod 0600 swap.<replaceable>192.168.4.6</replaceable></userinput>
- </screen>
+ <screen>
+ &prompt.root; <userinput>mkdir <replaceable>/netswapvolume/netswap</replaceable></userinput>
+ &prompt.root; <userinput>cd <replaceable>/netswapvolume/netswap</replaceable></userinput>
+ &prompt.root; <userinput>dd if=/dev/zero bs=1024 count=<replaceable>64000</replaceable> of=swap.<replaceable>192.168.4.6</replaceable></userinput>
+ &prompt.root; <userinput>chmod 0600 swap.<replaceable>192.168.4.6</replaceable></userinput>
+ </screen>
<para><replaceable>192.168.4.6</replaceable> is the IP address
for the diskless client.</para>
</step>
@@ -2874,13 +3065,31 @@
<para>On the NFS swap file server, add the following line to
<filename>/etc/exports</filename>:</para>
<programlisting>
- <replaceable>/netswapvolume</replaceable> -maproot=0:10 -alldirs <replaceable>margaux</replaceable>
+<replaceable>/netswapvolume</replaceable> -maproot=0:10 -alldirs <replaceable>margaux</replaceable>
</programlisting>
<para>Then tell <application>mountd</application> to reread the
exports file, as above.</para>
</step>
</procedure>
+ </sect4>
+
+ <sect4>
+ <title>NFS swap with FreeBSD 5.x</title>
+
+ <para>The kernel does not support enabling NFS swap at boot
+ time. Swap must be enabled by the startup scripts, by
+ mounting a writeable file system and creating and enabling a
+ swap file. This can be done, for example, in
+ <filename>/etc/rc.local</filename>. Example:</para>
+
+ <screen>
+ swapfile=/writableVolume/swap/swap.<replaceable>MyIP</replaceable>
+ dd if=/dev/zero of=$swapfile bs=1k count=1 oseek=100000
+ swapon $swapfile
+ rm -rf $swapfile
+ </screen>
+ </sect4>
</sect3>
<sect3>
@@ -2915,7 +3124,7 @@
to create the correct device entries (FreeBSD 5.0 and later
use &man.devfs.5; to allocate device nodes transparently for
the user, running <command>MAKEDEV</command> on these
- versions is useless).</para>
+ versions is pointless).</para>
</sect4>
More information about the freebsd-doc
mailing list