Booting off raidz *mostly* work
Adam McDougall
mcdouga9 at egr.msu.edu
Fri Sep 25 23:04:43 UTC 2009
Ollivier Robert wrote:
> According to Ollivier Robert:
>> By popular demand:
>> http://www.keltia.net/howtos/zfsboot
>
> I've succeeded in booting off raidz!
>
> There is a remaining issue with /boot/loader getting zfs errors at boot time
> resulting in loader.conf not being read. After setting the various variables
> manually (vm.kmem_size, vfs.root.mountfrom) and loading zfs, it works...
>
> http://static.keltia.net/download/raidz-boot.png
>
> I've updated the howto to reflect that.
>
> Any help in tackling the last issue is welcome.
>
I don't have help for raid-z, but I can contribute some parts of my own
procedure that have been helpful:
--------------------------------------------------------------------------
- If you would prefer to SSH into the livefs shell (have network access
but hate typing out the commands manually, or you have to put up with a
painful console terminal such as serial, not to mention that ctrl-c
makes the livefs shell abort and try to go back to sysinstall):
ifconfig foo0 inet 10.0.0.2/24
route add default 10.0.0.1
echo "nameserver 10.0.0.1" >> /etc/resolv.conf
echo "domain example.com" >> /etc/resolv.conf
mkdir /etc/ssh
ssh-keygen -t rsa1 -b 1024 -f /etc/ssh/ssh_host_key -N ""
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ""
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ""
cp /dist/etc/ssh/sshd_config /tmp/
echo "PermitRootLogin yes" >> /tmp/sshd_config
env > /tmp/env
ln -s /mnt2/bin/csh /bin/csh
/mnt2/usr/sbin/sshd -Ddf /tmp/sshd_config
From the network:
ssh root at 10.0.0.2
sh
. /tmp/env
. /mnt2/.profile
(I used /tmp/env to get the PATH settings from the sysinstall shell
propagated to the ssh shell, and I specifically used -Dd so you can only
login once since root has no password!)
----------------------------------------------------------------------------
- Thinking ahead to when I need to reinstall a system in the future
(happens many times during testing), I need to remember to wipe out
existing gmirror or partitions/partition tables:
gmirror clear /dev/da0p2 /dev/da1p2
gpart delete -i 3, 2, 1, gpart destroy da0 etc
I strongly suggest putting swap on a gmirror if it isn't going to be
protected by another form of raid, even though it contains no valuable
data, I don't want my system to go down because of a failure on a
non-redundant swap volume.
Something like:
gmirror label -v -b prefer swap /dev/da0p2 /dev/da1p2 && \
echo '/dev/mirror/swap none swap sw 0 0' >> /rpool/ROOT/freebsd/etc/fstab
It would also be handy to enable gmirror status checks in periodic.conf
although I used a cronned script so I get paged more than once a day if
there is a problem.
---------------------------------------------------------------------
- You could use jot to duplicate various gpart commands to multiple
drives, examples:
Clear 2 drives with up to 3 partitions each:
jot -w "gpart delete -i 3 da" -s \; 2 0 1 | sh
jot -w "gpart delete -i 2 da" -s \; 2 0 1 | sh
jot -w "gpart delete -i 1 da" -s \; 2 0 1 | sh
jot -w "gpart destroy da" -s \; 2 0 1 | sh
Clear 8 drives with up to 3 partitions each:
jot -w "gpart delete -i 3 da" -s \; 8 0 7 | sh
jot -w "gpart delete -i 2 da" -s \; 8 0 7 | sh
jot -w "gpart delete -i 1 da" -s \; 8 0 7 | sh
jot -w "gpart destroy da" -s \; 8 0 7 | sh
-----------------------------------------------------------------------
- In 8.0-BETA2 or above, you can use some new features in gpart for gpt
auto sizing to automatically calculate the starting sector or even let
your last partition use all remaining space: (helpful so you don't need
to do as much math and less typing)
Create gpt, set partition active, create 3 partitions on 2 drives using
all remaining space after the boot and swap:
jot -w "gpart create -s GPT da" -s \; 2 0 1 | sh
jot -w "echo 'a 1' | fdisk -f - -q /dev/da" -s \; 2 0 1 | sh
jot -w "gpart add -s 128 -t freebsd-boot da" -s \; 2 0 1 | sh
jot -w "gpart add -s 4194304 -t freebsd-swap da" -s \; 2 0 1 | sh
jot -w "gpart add -t freebsd-zfs da" -s \; 2 0 1 | sh
Similar for installing boot code for 2 drives:
jot -w "gpart bootcode -b /rpool/ROOT/freebsd/boot/pmbr -p \
/rpool/ROOT/freebsd/boot/gptzfsboot -i 1 da" -s \; 2 0 1 | sh
------------------------------------------------------------------------
- Instead of extracting dists from the install media, I chose to build
tar files of a system image using procedures gleaned from the jail
manpage (its pretty easy), that way I can add loader support and plenty
of customizations to my image, use an up to date build, even include the
source and ports trees for convenience into my tar, and the tar tends to
extract in just a couple minutes on the destination system (70-180
seconds depending on the system and what is in the tar). This way I can
also use the 8.0 livefs to install either 7.x or 8.x just based on which
tar I download. Obviously this has more benefit if you have more than a
small number of systems to install; if its just one or two then
pre-customizing isn't worth it. I stopped including the portsnap files
dir in favor of a post-install fresh 'portsnap fetch' though, because
after just a couple weeks, it seemed like my 'portsnap fetch' and
'update' took longer than getting a fresh copy. Since I track -stable
on my servers, its nice to keep my tar updated every month or few weeks
and then my new installs don't each require a source update since they
start life being fairly up to date already. Also a very nice result is
by adding NO_FOO options to the image build, the resulting image never
has parts of the base that you never wanted in the first place, for
example if you leave out sendmail or lpr, so you don't have to worry
about 'make check-old' being complete enough.
More information about the freebsd-current
mailing list