Root on ZFS RAIDZ1
Kurt Lidl
lidl at pix.net
Fri May 4 13:30:39 UTC 2018
I have been using various sparc64 machines with a ZFS only installation
for quite a while, and found that it works reliably for me. My earlier
machines were Netra-T1 (all retired now), and currently I have both
a V240 and three V120 machines using this configuration.
Here are my notes that I use during the partitioning phase of the
install. (ie, start a normal install as discussed in the web page
you mention, and during the 'Shell' method of partitioning, I run
these commands):
You may or may not want to execute the commented out sysctl,
which will force the ZFS installation to use 4096 byte blocks.
Given that all the disks I've ever used on sparc64 are really
512byte block devices, I'm not sure if this is a great idea or
not. I typically do run with 4096 byte blocks.
-----------------------------------------------------------------
kldload zfs
# force 4K minimum zfs block size
#sysctl vfs.zfs.min_auto_ashift=12
# appropriate for a 36GB disk w/4+GB of swap
zfssize=30000M
# appropriate for a 72GB disk w/4+GB of swap
zfssize=65500M
# appropriate for a 72GB disk w/8+GB of swap
zfssize=61500M
disks="da0 da1"
for d in $disks
do
gpart destroy -F $d
gpart create -s VTOC8 $d
gpart add -t freebsd-zfs -s $zfssize $d
gpart add -t freebsd-swap $d
gpart bootcode -p /boot/zfsboot $d
dd if=/boot/zfsloader of=/dev/${d}a bs=512 oseek=1024 \
conv=notrunc,sync
done
# note, we use -R /mnt to set a different root mountpoint
P=zroot
mnt=/mnt
zpool create -f -o cachefile=none -O checksum=fletcher4 \
-R $mnt -m none $P mirror /dev/da0a /dev/da1a
zfs create $P/ROOT
zfs create $P/ROOT/default
zpool set bootfs=$P/ROOT/default $P
# because of "-R /mnt" earlier, this is "/mnt" effectively for this session
zfs set mountpoint=/ $P/ROOT/default
zfs create $P/vartmp
zfs create $P/usrports
zfs create $P/usrsrc
zfs create $P/usrobj
zfs create $P/home
zfs set mountpoint=/var/tmp $P/vartmp
zfs set mountpoint=/home $P/home
zfs set mountpoint=/usr/ports $P/usrports
zfs set mountpoint=/usr/src $P/usrsrc
zfs set mountpoint=/usr/obj $P/usrobj
zfs mount -a
gmirror label -b prefer gswap0 da0b da1b
eeprom boot-device="disk0 disk1"
exit
-----------------------------------------------------------------
At the end of the installation process, you will need to fix up the
/etc/rc.conf and /boot/loader.conf too:
# Add to /etc/rc.conf:
clear_tmp_enable="YES"
ntpdate_enable="YES"
rwhod_enable="YES"
rwhod_flags="-m"
zfs_enable="YES"
# Add to /boot/loader.conf:
zfs_load="YES"
geom_mirror_load="YES"
vfs.zfs.prefetch_disable=1
# Add to /etc/fstab:
# Device Mountpoint FStype Options Dmp Pass#
/dev/mirror/gswap none swap sw 0 0
#md /tmp mfs rw,-s64m 0 0
tmpfs /tmp tmpfs rw,size=64m,mode=1777 0 0
# turn off ttyv? entries in /etc/ttys
# turn off ttyu2 entry in /etc/ttys
I hope this helps you get the system working well.
Best wishes,
-Kurt
On 5/4/18 5:08 AM, iwama at t3.rim.or.jp wrote:
> I want Sun Fire V245 to use FreeBSD-11.1-RELEASE with Root on ZFS RAIDZ1.
> The installation procedure of Root on ZFS is as follow page:
>
> https://wiki.freebsd.org/RootOnZFS/VTOC8ZFSBoot/9.x-RELEASE
>
> I finished the installation without any problem, but it was some problems
> at reboot.
>
> 1) It do not boot at all
> 2) boot takes a very long time, more than 3 HOURS
> 3) It takes some time for boot, about 10 minutes
>
> I tried RAIDZ1 with 9.3-RELEASE, RAIDZ2, RAIDZ10 and mirror with
> 11.1-RELEASE,
> the results were as follows, da0-3 are the name of boot disk. (Fire V245 has
> four disks):
>
> RAIDZ1(zpool create -o altroot=/mnt -O canmount=off zroot raidz1 da0a da1a
> da2a da3a)
> da0 da1 da2 da3
> 11.1-RELEASE NG NG SLOW SLOW
> 9.3-RELEASE NG NG SLOW SLOW
>
> RAIDZ2(zpool create -o altroot=/mnt -O canmount=off zroot raidz2 da0a da1a
> da2a da3a)
> da0 da1 da2 da3
> 11.1-RELEASE SLOW SLOW SLOW SLOW
>
> RAIDZ10(zpool create -o altroot=/mnt -O canmount=off zroot mirror da0a da1a
> mirror da2a da3a)
> da0 da1 da2 da3
> 11.1-RELEASE slow slow slow slow
>
> mirror(zpool create -o altroot=/mnt -O canmount=off zroot mirror da0a da1a
> da2a da3a)
> da0 da1 da2 da3
> 11.1-RELEASE OK OK OK OK
>
> OK: OK(boot at normal speed), NG: 1), SLOW: 2), slow: 3)
>
>
> From the above results and the logs below, there is no problem with
> mirror, but it is a big
> problem in RAIDZ1 or RAIDZ2.
> Has anyone been running Root on RAIDZ1 on the sparc64 platform?
>
>
> Install steps(only the part entered manually):
> ===
> # zpool labelclear -f /dev/da0a
> # zpool labelclear -f /dev/da1a
> # zpool labelclear -f /dev/da2a
> # zpool labelclear -f /dev/da3a
> #
> # sysctl kern.disks
> kern.disks: cd0 da3 da2 da1 da0
> # gpart destroy -F da0
> da0 destroyed
> # gpart destroy -F da1
> da1 destroyed
> # gpart destroy -F da2
> da2 destroyed
> # gpart destroy -F da3
> da3 destroyed
> # gpart create -s VTOC8 da0
> da0 created
> # gpart create -s VTOC8 da1
> da1 created
> # gpart create -s VTOC8 da2
> da2 created
> # gpart create -s VTOC8 da3
> da3 created
> #
> # gpart add -s 270G -t freebsd-zfs da0
> da0a added
> # gpart add -s 270G -t freebsd-zfs da1
> da1a added
> # gpart add -s 270G -t freebsd-zfs da2
> da2a added
> # gpart add -s 270G -t freebsd-zfs da3
> da3a added
> # gpart add -t freebsd-swap da0
> da0b added
> # gpart add -t freebsd-swap da1
> da1b added
> # gpart add -t freebsd-swap da2
> da2b added
> # gpart add -t freebsd-swap da3
> da3b added
> #
> ***The following lines differ depending on the RAID method
> # zpool create -o altroot=/mnt -O canmount=off zroot raidz1 da0a da1a da2a
> da3a
> #
> # zpool export zroot
> # gpart bootcode -p /boot/zfsboot da0
> # gpart bootcode -p /boot/zfsboot da1
> # gpart bootcode -p /boot/zfsboot da2
> # gpart bootcode -p /boot/zfsboot da3
> # sysctl kern.geom.debugflags=0x10
> kern.geom.debugflags: 0 -> 16
> # dd if=/boot/zfsloader of=/dev/da0a bs=512 oseek=1024 conv=notrunc,sync
> 466+1 records in
> 467+0 records out
> 239104 bytes transferred in 2.845066 secs (84042 bytes/sec)
> # dd if=/boot/zfsloader of=/dev/da1a bs=512 oseek=1024 conv=notrunc,sync
> 466+1 records in
> 467+0 records out
> 239104 bytes transferred in 2.832603 secs (84411 bytes/sec)
> # dd if=/boot/zfsloader of=/dev/da2a bs=512 oseek=1024 conv=notrunc,sync
> 466+1 records in
> 467+0 records out
> 239104 bytes transferred in 2.815799 secs (84915 bytes/sec)
> # dd if=/boot/zfsloader of=/dev/da3a bs=512 oseek=1024 conv=notrunc,sync
> 466+1 records in
> 467+0 records out
> 239104 bytes transferred in 2.828102 secs (84546 bytes/sec)
> # zpool import -o altroot=/mnt zroot
> #
> # zfs set checksum=fletcher4 zroot
> # zfs set atime=off zroot
> #
> # zfs create -o mountpoint=/ zroot/ROOT
>
> #
> # zpool set bootfs=zroot/ROOT zroot
> #
> # cat << EOF > /tmp/bsdinstall_etc/fstab
>> # Device Mountpoint FStype Options Dump Pass#
>> /dev/da0b none swap sw 0 0
>> /dev/da1b none swap sw 0 0
>> /dev/da2b none swap sw 0 0
>> /dev/da3b none swap sw 0 0
>> ^D#
> #
> # exit
>
>
> # echo 'zfs_enable="YES"' >> /etc/rc.conf
> #
> # cat /boot/loader.conf
> zfs_load="YES"
> ***Except for 9.3-RELEASE, since the line of "zfs_load="YES"" is originally
> in
> ***/boot/loader.conf, addition is unnecessary
> #
> # zpool set cachefile=/boot/zfs/zpool.cache zroot
> # exit
>
> ***In 11.1-RELEASE, the last "zpool set cachefile=/boot/zfs/zpool.cache
> zroot" failed,
> shown as "internal error: failed to initialize ZFS library"
> ===
>
>
> NOT boot log:
> ===
> {1} ok boot disk0
>
> SC Alert: Host System has Reset
> Probing system devices
> Probing memory
> Probing I/O buses
> screen not found.
> keyboard not found.
> Keyboard not present. Using ttya for input and output.
> Probing system devices
> Probing memory
> Probing I/O buses
>
>
> Sun Fire V245, No Keyboard
> Copyright 2006 Sun Microsystems, Inc. All rights reserved.
> OpenBoot 4.22.22, 16384 MB memory installed, Serial #XXXXXXXX.
> Ethernet address 0:14:4f:XX:XX:XX, Host ID: XXXXXXXX.
>
>
>
> Initializing 1MB of memory at addr 137ff0a000
> Initializing 1MB of memory at addr 137ff06000
> Initializing 1MB of memory at addr 137ff00000
> Initializing 1MB of memory at addr 137fee0000
> Initializing 1MB of memory at addr 137fe90000
> Initializing 1MB of memory at addr 137fdea000
> Initializing 14MB of memory at addr 137f000000
> Initializing 2032MB of memory at addr 1300000000
> Initializing 2048MB of memory at addr 1200000000
> Initializing 2048MB of memory at addr 1100000000
> Initializing 2048MB of memory at addr 1000000000
> Initializing 2048MB of memory at addr 300000000
> Initializing 2048MB of memory at addr 200000000
> Initializing 2048MB of memory at addr 100000000
> Initializing 2048MB of memory at addr 0
> Rebooting with command: boot disk0
> Boot device: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 0,0 File
> and args:
>
>>> FreeBSD/sparc64 ZFS boot block
> Boot path: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 0,0:a
> Consoles: Open Firmware console
> ZFS: i/o error - all block copies unavailable
> ZFS: can't read MOS object directory
> ZFS: can't find root filesystem
>
> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0
> (Fri Jul 21 08:44:42 UTC 2017 root at releng2.nyi.freebsd.org)
> bootpath=""
>
> can't load 'kernel'
>
> Type '?' for a list of commands, 'help' for more detailed help.
> OK reboot
> Rebooting...
> Program terminated
> {1} ok
> ===
>
>
> VERY SLOW boot log:
> ===
> {1} ok boot disk3
>
> SC Alert: Host System has Reset
> Probing system devices
> Probing memory
> Probing I/O buses
> screen not found.
> keyboard not found.
> Keyboard not present. Using ttya for input and output.
> Probing system devices
> Probing memory
> Probing I/O buses
>
>
> Sun Fire V245, No Keyboard
> Copyright 2006 Sun Microsystems, Inc. All rights reserved.
> OpenBoot 4.22.22, 16384 MB memory installed, Serial #XXXXXXXX.
> Ethernet address 0:14:4f:XX:XX:XX, Host ID: XXXXXXXX.
>
>
>
> Initializing 1MB of memory at addr 137ff0a000
> Initializing 1MB of memory at addr 137ff06000
> Initializing 1MB of memory at addr 137ff00000
> Initializing 1MB of memory at addr 137fee0000
> Initializing 1MB of memory at addr 137fe90000
> Initializing 1MB of memory at addr 137fdea000
> Initializing 14MB of memory at addr 137f000000
> Initializing 2032MB of memory at addr 1300000000
> Initializing 2048MB of memory at addr 1200000000
> Initializing 2048MB of memory at addr 1100000000
> Initializing 2048MB of memory at addr 1000000000
> Initializing 2048MB of memory at addr 300000000
> Initializing 2048MB of memory at addr 200000000
> Initializing 2048MB of memory at addr 100000000
> Initializing 2048MB of memory at addr 0
> Rebooting with command: boot disk3
> Boot device: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 3,0 File
> and args:
>
>>> FreeBSD/sparc64 ZFS boot block
> Boot path: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 3,0:a
> Consoles: Open Firmware console
>
>
> ***Slow from here(\|/-\|/- animation)***
> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0
> (Fri Jul 21 08:44:42 UTC 2017 root at releng2.nyi.freebsd.org)
> bootpath="zfs:zroot/ROOT:"
> Loading /boot/defaults/loader.conf
> /boot/kernel/kernel data=0xc47740+0xf4078 syms=[0x8+0xd5a38+0x8+0xc746a]
> /boot/entropy size=0x1000
> /boot/kernel/zfs.ko text=0x25cc60 data=0xb9d0+0x90050
> |syms=[0x8+0x1bd38/-+0x8+0x16db2]
> loading required module 'opensolaris'
> /boot/kernel/opensolaris.ko text=0x3730 data=0x2c8+0x1a30
> syms=[0x8+0xe10+0x8+0x968]
> ***It is slow until the above is displayed.***
>
>
> Hit [Enter] to boot immediately, or any other key for command prompt.
> Booting [/boot/kernel/kernel] in 9 seconds... Booting [/boot/kernel/kernel]
> in 8 seconds... Booting [/boot/kernel/kernel] in 7 seconds... Booting
> [/boot/kernel/kernel] in 6 seconds... Booting [/boot/kernel/kernel] in 5
> seconds... Booting [/boot/kernel/kernel] in 4 seconds... Booting
> [/boot/kernel/kernel] in 3 seconds... Booting [/boot/kernel/kernel] in 2
> seconds... Booting [/boot/kernel/kernel] in 1 second... Booting
> [/boot/kernel/kernel]...
> -\|/-\|/-\|/-\|/-\|jumping to kernel entry at 0xc00b0000.
> ===
>
> ---
> Yoshihiko Iwama
>
>
> _______________________________________________
> freebsd-sparc64 at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
> To unsubscribe, send any mail to "freebsd-sparc64-unsubscribe at freebsd.org"
>
More information about the freebsd-sparc64
mailing list