Using r/o root with amd(8)-mounted file systems

David Wolfskill david at catwhisker.org
Wed Nov 5 07:56:24 PST 2008


In networks that I control and which are "sufficiently small" while
having enough resources to make it practical -- such as at home -- I
like to do a few things to split up the workload and make the "common
case" (of merely quietly doing their jobs) easier for the bulk of the
machines ... at the expense of needing to tweak things a bit initially
to get there, and needing to do a bit more work for upgrades.

For example, one of the things I like to do is set up "production"
machines (e.g., my firewall box and the central mail server) so they:

* Each have 2 separate bootable slices, each of which contains a
  fully-functional root on the "a" partition and /usr on the "d"
  partition, and a 3rd slice to contain "everything else" (that is
  used regardless of which slice is the current boot slice): swap
  space, /var, and a file system that contains the directories where
  the /home and /usr/local symlinks point.  (Yes, I make /usr/local
  a symlink.)  Because I can easily control which slice is the default
  boot slice via boot0cfg(8), I use the FreeBSD boot loader.

* Use NIS for "installation-wide" notions of users & groups.  (Hey; one
  of the machines at home is a SPARCstation 5/170, after all.)

* Use NFS for making certain file systems & directories have an
  "appearance" on the local machine.  (Home directories & a few others
  are presently served by the above-cited SS5/170, though I've started
  lobbying the "family CFO" to free up funds to migrate that job to a
  ReadyNAS.  /usr/{obj,ports,src} are hosted on the build machine.)

* Avoid "hard" NFS mounts.  I use amd(8) to manage the NFS mounts, and
  it's been working well for me for around a decade or so.

* Do not have their own /usr/src, /usr/obj, or /usr/ports directory
  hierarchies.  Rather, these are NFS-mounted from a dedicated "build
  machine" that has no role in the usual day-to-day "production"
  activities.  the build machine has a local private mirror of the
  FreeBSD CVS repository which I update in 2 stages overnight (via
  cron(8), of course), and I track branches of interest on it, usually
  daily, as well as update ports on it daily.  At present, I'm tracking
  RELENG_6, RELENG_7, and HEAD.

  Thus, the build machine, in addition to building the "world"
  (userland) and its own kernel, also builds kernels for the other
  machines.

* Mount /usr read-only.  Yes, this becomes a slight nuisance when it's
  time to upgrade, but that nearly vanishes inside a few csh(1) aliases.
  It's slightly more annoying when it's time to upgrade ports on
  production machines, but I still find it useful:  it provides a degree
  of assurance that things aren't likely changing without my knowledge.
  And should there be a reboot, that's one more file system that need
  not be checked.  (And there have been cases where the UPS batteries
  haven't lasted as long as an electrical supply outage.)

The above all have been working well for me -- as long as I've had a
working build machine, anyway.

I had tried mounting the root file system read-only (back in 3.x days);
while it mostly worked, sshd(8) threw a bit of a hissy-fit because it
couldn't chown(1) a pty entry in /dev.  And since my normal mode of
operation is to access everything from my laptop (running FreeBSD, of
course) vis ssh(1), I wasn't too keen on risking running afoul of
sshd(8).  :-}

Now that /dev is merely a figment of the kernel's imagination :-}, I
thought I'd re-try mounting root as read-only.  As expected, sshd(8)
didn't complain -- at least, not about ownership of a pty.

What I did encounter -- at least sometimes -- is that If I specify that
/ is read-only in /etc/fstab, on reboot:

* sometimes everything work nicely.

* other times, the interaction between the read-only root and amd(8) is
  such that amd(8) is started, but doesn't actually work.  In such
  cases, a workaround is to mount root read-write, restart amd(8), then
  mount root read-only.

I'm a bit bothered by the nuisance of the latter, but even more
concerned about the apparent lack of determinism in the process.

Any ideas on how to track this down?

The most recent occurrence was on a machine I'm in the process of
setting up to replace our internal mail server:

albert(7.1-P)[1] uname -a
FreeBSD albert.catwhisker.org 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #1: Wed Nov  5 05:31:00 PST 2008     root at freebeast.catwhisker.org:/common/S3/obj/usr/src/sys/ALBERT  i386
albert(7.1-P)[2] 

I rebooted it about 5 times in succession with amd(8) failing to do its
job; on the first & the last of these, I performed the above-cited
"workaround", after which a reboot came up "normally":

albert(7.1-P)[2] mount
/dev/ad0s2a on / (ufs, local, read-only, soft-updates)
devfs on /dev (devfs, local)
/dev/ad0s2d on /usr (ufs, NFS exported, local, read-only)
/dev/ad0s3d on /var (ufs, local, soft-updates)
/dev/ad0s3e on /bkp (ufs, local, soft-updates)
/dev/ad1s1d on /common (ufs, local, soft-updates)
/dev/md0 on /tmp (ufs, asynchronous, local)
pid660 at albert:/host on /host (nfs)
pid660 at albert:/net on /net (nfs)
pogo:/cdrom on /.amd_mnt/pogo/host/cdrom (nfs, nosuid)
pogo:/export on /.amd_mnt/pogo/host/export (nfs, nosuid)
pogo:/export/bd1 on /.amd_mnt/pogo/host/export/bd1 (nfs, nosuid)
pogo:/export/bd2 on /.amd_mnt/pogo/host/export/bd2 (nfs, nosuid)
pogo:/export/home on /.amd_mnt/pogo/host/export/home (nfs, nosuid)
pogo:/export/local on /.amd_mnt/pogo/host/export/local (nfs, nosuid)
albert(7.1-P)[3] uptime
 7:29AM  up 17 mins, 1 user, load averages: 0.00, 0.00, 0.01
 albert(7.1-P)[4] 

Deploying the machine in production is neither urgent nor critical at
this point, so I have some time to work on it.

Here's where rcorder(8) has to say:

albert(7.1-P)[3] rcorder /etc/rc.d/* /usr/local/etc/rc.d/*
/etc/rc.d/dumpon
/etc/rc.d/ddb
/etc/rc.d/initrandom
/etc/rc.d/geli
/etc/rc.d/gbde
/etc/rc.d/encswap
/etc/rc.d/ccd
/etc/rc.d/swap1
/etc/rc.d/early.sh
/etc/rc.d/fsck
/etc/rc.d/root
/etc/rc.d/hostid
/etc/rc.d/mdconfig
/etc/rc.d/mountcritlocal
/etc/rc.d/zfs
/etc/rc.d/FILESYSTEMS
/etc/rc.d/var
/etc/rc.d/cleanvar
/etc/rc.d/random
/etc/rc.d/adjkerntz
/etc/rc.d/atm1
/etc/rc.d/hostname
/etc/rc.d/ipfilter
/etc/rc.d/ipnat
/etc/rc.d/ipfs
/etc/rc.d/kldxref
/etc/rc.d/sppp
/etc/rc.d/addswap
/etc/rc.d/auto_linklocal
/etc/rc.d/sysctl
/etc/rc.d/serial
/etc/rc.d/netif
/etc/rc.d/ip6addrctl
/etc/rc.d/atm2
/etc/rc.d/pfsync
/etc/rc.d/pflog
/etc/rc.d/pf
/etc/rc.d/isdnd
/etc/rc.d/ppp
/etc/rc.d/routing
/etc/rc.d/ip6fw
/etc/rc.d/network_ipv6
/etc/rc.d/devd
/etc/rc.d/ipsec
/etc/rc.d/ipfw
/etc/rc.d/nsswitch
/etc/rc.d/resolv
/etc/rc.d/mroute6d
/etc/rc.d/route6d
/etc/rc.d/mrouted
/etc/rc.d/routed
/etc/rc.d/netoptions
/etc/rc.d/NETWORKING
/etc/rc.d/mountcritremote
/etc/rc.d/ldconfig
/etc/rc.d/tmp
/etc/rc.d/cleartmp
/usr/local/etc/rc.d/xfs
/usr/local/etc/rc.d/xdm.sh.noauto
/usr/local/etc/rc.d/rplayd.sh.sample
/etc/rc.d/accounting
/etc/rc.d/devfs
/etc/rc.d/ipmon
/etc/rc.d/mdconfig2
/etc/rc.d/newsyslog
/etc/rc.d/syslogd
/etc/rc.d/savecore
/etc/rc.d/archdep
/etc/rc.d/abi
/etc/rc.d/SERVERS
/etc/rc.d/named
/etc/rc.d/ntpdate
/etc/rc.d/rpcbind
/etc/rc.d/nfsclient
/etc/rc.d/nisdomain
/etc/rc.d/ypserv
/etc/rc.d/ypbind
/etc/rc.d/amd
/etc/rc.d/atm3
/etc/rc.d/auditd
/etc/rc.d/dmesg
/etc/rc.d/ipxrouted
/etc/rc.d/kerberos
/etc/rc.d/kadmind
/etc/rc.d/keyserv
/etc/rc.d/kpasswdd
/etc/rc.d/quota
/etc/rc.d/nfsserver
/etc/rc.d/mountd
/etc/rc.d/nfsd
/etc/rc.d/statd
/etc/rc.d/lockd
/etc/rc.d/pppoed
/etc/rc.d/pwcheck
/etc/rc.d/virecover
/etc/rc.d/DAEMON
/etc/rc.d/apm
/etc/rc.d/apmd
/etc/rc.d/bootparams
/etc/rc.d/hcsecd
/etc/rc.d/bthidd
/etc/rc.d/local
/etc/rc.d/lpd
/etc/rc.d/motd
/etc/rc.d/mountlate
/etc/rc.d/nscd
/etc/rc.d/ntpd
/etc/rc.d/powerd
/etc/rc.d/rarpd
/etc/rc.d/sdpd
/etc/rc.d/rfcomm_pppd_server
/etc/rc.d/rtadvd
/etc/rc.d/rwho
/etc/rc.d/timed
/etc/rc.d/ugidfw
/etc/rc.d/yppasswdd
/etc/rc.d/LOGIN
/usr/local/etc/rc.d/mysql-server
/usr/local/etc/rc.d/htcacheclean
/usr/local/etc/rc.d/dbus
rcorder: requirement `usbd' in file `/usr/local/etc/rc.d/hald' has no providers.
/usr/local/etc/rc.d/hald
/usr/local/etc/rc.d/ffserver
/usr/local/etc/rc.d/apache22
/etc/rc.d/ypxfrd
/etc/rc.d/ypupdated
/etc/rc.d/ypset
/etc/rc.d/wpa_supplicant
/etc/rc.d/watchdogd
/etc/rc.d/syscons
/etc/rc.d/sshd
/etc/rc.d/sendmail
/etc/rc.d/cron
/etc/rc.d/jail
/etc/rc.d/localpkg
/etc/rc.d/securelevel
/etc/rc.d/power_profile
/etc/rc.d/othermta
/etc/rc.d/natd
/etc/rc.d/msgs
/etc/rc.d/moused
/etc/rc.d/mixer
/etc/rc.d/inetd
/etc/rc.d/idmapd
/etc/rc.d/hostapd
/etc/rc.d/geli2
/etc/rc.d/ftpd
/etc/rc.d/ftp-proxy
/etc/rc.d/dhclient
/etc/rc.d/bsnmpd
/etc/rc.d/bridge
/etc/rc.d/bluetooth
/etc/rc.d/bgfsck
albert(7.1-P)[4] 

Peace,
david
-- 
David H. Wolfskill				david at catwhisker.org
Depriving a girl or boy of an opportunity for education is evil.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20081105/f50b2a20/attachment.pgp


More information about the freebsd-stable mailing list