Root on ZFS RAIDZ1

Kurt Lidl lidl at pix.net
Fri May 4 21:18:12 UTC 2018


I have my V240 configured to have one set of mirrored
drives on the first two disks, and a second set of mirrored
disks on the last two disks.

This allows me to have two completely independent OS installs
on the machine, and easily switch between them, depending on
which disk I boot the system from.  I use different ZFS pool
names on the different mirrors, so I can, if needed, import
the ZFS pool from the other set of disks.  Handy for fixing
up certain classes of experimental errors on the machine.

For a while, I ran with one set of mirrored disks, and just
had non-mirrored zpools set up on the other two disks, one
mounted as /usr/src and the other mounted as /usr/obj.  That
experiment didn't show any huge benefit, so I just went to
my current setup, with two sets of independent mirrored disks.

I never tried a RAIDZ of all four disks on the sparc64.

-Kurt

On 5/4/18 5:09 PM, iwama at t3.rim.or.jp wrote:
> Hi Kurt,
> 
>    Thank you for your reply.
> 
>    Thank you for providing your achievements and procedures. I will try it in
> your procedure.
> 
>    Please let me ask a question.
>    In your procedure, the RAID method is mirror, but is there a record of
> RAIDZ of 4 disks
> with V240?
> 
> Regards,
> ---
> Yoshihiko Iwama
> 
> -----Original Message-----
> From: owner-freebsd-sparc64 at freebsd.org <owner-freebsd-sparc64 at freebsd.org>
> On Behalf Of Kurt Lidl
> Sent: Friday, May 4, 2018 10:31 PM
> To: freebsd-sparc64 at freebsd.org
> Subject: Re: Root on ZFS RAIDZ1
> 
> I have been using various sparc64 machines with a ZFS only installation for
> quite a while, and found that it works reliably for me.  My earlier machines
> were Netra-T1 (all retired now), and currently I have both a V240 and three
> V120 machines using this configuration.
> 
> Here are my notes that I use during the partitioning phase of the install.
> (ie, start a normal install as discussed in the web page you mention, and
> during the 'Shell' method of partitioning, I run these commands):
> 
> You may or may not want to execute the commented out sysctl, which will
> force the ZFS installation to use 4096 byte blocks.
> Given that all the disks I've ever used on sparc64 are really 512byte block
> devices, I'm not sure if this is a great idea or not.  I typically do run
> with 4096 byte blocks.
> 
> -----------------------------------------------------------------
> 
> kldload zfs
> 
> # force 4K minimum zfs block size
> #sysctl vfs.zfs.min_auto_ashift=12
> 
> # appropriate for a 36GB disk w/4+GB of swap zfssize=30000M # appropriate
> for a 72GB disk w/4+GB of swap zfssize=65500M # appropriate for a 72GB disk
> w/8+GB of swap zfssize=61500M
> 
> disks="da0 da1"
> for d in $disks
> do
>           gpart destroy -F $d
>           gpart create -s VTOC8 $d
>           gpart add -t freebsd-zfs -s $zfssize $d
>           gpart add -t freebsd-swap $d
>           gpart bootcode -p /boot/zfsboot $d
>           dd if=/boot/zfsloader of=/dev/${d}a bs=512 oseek=1024 \
> 		conv=notrunc,sync
> done
> 
> # note, we use -R /mnt to set a different root mountpoint P=zroot mnt=/mnt
> zpool create -f -o cachefile=none -O checksum=fletcher4 \
>           -R $mnt -m none $P mirror /dev/da0a /dev/da1a
> 
> zfs create $P/ROOT
> zfs create $P/ROOT/default
> zpool set bootfs=$P/ROOT/default $P
> 
> # because of "-R /mnt" earlier, this is "/mnt" effectively for this session
> zfs set mountpoint=/ $P/ROOT/default
> 
> zfs create $P/vartmp
> zfs create $P/usrports
> zfs create $P/usrsrc
> zfs create $P/usrobj
> zfs create $P/home
> zfs set mountpoint=/var/tmp $P/vartmp
> zfs set mountpoint=/home $P/home
> zfs set mountpoint=/usr/ports $P/usrports zfs set mountpoint=/usr/src
> $P/usrsrc zfs set mountpoint=/usr/obj $P/usrobj zfs mount -a
> 
> gmirror label -b prefer gswap0 da0b da1b
> 
> eeprom boot-device="disk0 disk1"
> 
> exit
> 
> -----------------------------------------------------------------
> 
> At the end of the installation process, you will need to fix up the
> /etc/rc.conf and /boot/loader.conf too:
> 
> # Add to /etc/rc.conf:
> clear_tmp_enable="YES"
> ntpdate_enable="YES"
> rwhod_enable="YES"
> rwhod_flags="-m"
> zfs_enable="YES"
> 
> # Add to /boot/loader.conf:
> zfs_load="YES"
> geom_mirror_load="YES"
> vfs.zfs.prefetch_disable=1
> 
> # Add to /etc/fstab:
> # Device                Mountpoint     FStype  Options Dmp     Pass#
> /dev/mirror/gswap       none           swap    sw              0       0
> #md                     /tmp           mfs     rw,-s64m        0       0
> tmpfs                   /tmp           tmpfs   rw,size=64m,mode=1777 0 0
> 
> # turn off ttyv? entries in /etc/ttys
> # turn off ttyu2 entry in /etc/ttys
> 
> I hope this helps you get the system working well.
> 
> Best wishes,
> 
> -Kurt
> 
> On 5/4/18 5:08 AM, iwama at t3.rim.or.jp wrote:
>>     I want Sun Fire V245 to use FreeBSD-11.1-RELEASE with Root on ZFS
> RAIDZ1.
>> The installation procedure of Root on ZFS is as follow page:
>>
>> https://wiki.freebsd.org/RootOnZFS/VTOC8ZFSBoot/9.x-RELEASE
>>
>>     I finished the installation without any problem, but it was some
>> problems at reboot.
>>
>> 1) It do not boot at all
>> 2) boot takes a very long time, more than 3 HOURS
>> 3) It takes some time for boot, about 10 minutes
>>
>>     I tried RAIDZ1 with 9.3-RELEASE, RAIDZ2, RAIDZ10 and mirror with
>> 11.1-RELEASE, the results were as follows, da0-3 are the name of boot
>> disk. (Fire V245 has four disks):
>>
>> RAIDZ1(zpool create -o altroot=/mnt -O canmount=off zroot raidz1 da0a
>> da1a da2a da3a)
>>                  da0  da1  da2  da3
>> 11.1-RELEASE  NG   NG   SLOW SLOW
>>    9.3-RELEASE  NG   NG   SLOW SLOW
>>
>> RAIDZ2(zpool create -o altroot=/mnt -O canmount=off zroot raidz2 da0a
>> da1a da2a da3a)
>>                  da0  da1  da2  da3
>> 11.1-RELEASE  SLOW SLOW SLOW SLOW
>>
>> RAIDZ10(zpool create -o altroot=/mnt -O canmount=off zroot mirror da0a
>> da1a mirror da2a da3a)
>>                  da0  da1  da2  da3
>> 11.1-RELEASE  slow slow slow slow
>>
>> mirror(zpool create -o altroot=/mnt -O canmount=off zroot mirror da0a
>> da1a da2a da3a)
>>                  da0  da1  da2  da3
>> 11.1-RELEASE  OK   OK   OK   OK
>>
>> OK: OK(boot at normal speed), NG: 1), SLOW: 2), slow: 3)
>>
>>
>>     From the above results and the logs below, there is no problem with
>> mirror, but it is a big problem in RAIDZ1 or RAIDZ2.
>>     Has anyone been running Root on RAIDZ1 on the sparc64 platform?
>>
>>
>> Install steps(only the part entered manually):
>> ===
>> # zpool labelclear -f /dev/da0a
>> # zpool labelclear -f /dev/da1a
>> # zpool labelclear -f /dev/da2a
>> # zpool labelclear -f /dev/da3a
>> #
>> # sysctl kern.disks
>> kern.disks: cd0 da3 da2 da1 da0
>> # gpart destroy -F da0
>> da0 destroyed
>> # gpart destroy -F da1
>> da1 destroyed
>> # gpart destroy -F da2
>> da2 destroyed
>> # gpart destroy -F da3
>> da3 destroyed
>> # gpart create -s VTOC8 da0
>> da0 created
>> # gpart create -s VTOC8 da1
>> da1 created
>> # gpart create -s VTOC8 da2
>> da2 created
>> # gpart create -s VTOC8 da3
>> da3 created
>> #
>> # gpart add -s 270G -t freebsd-zfs da0 da0a added # gpart add -s 270G
>> -t freebsd-zfs da1 da1a added # gpart add -s 270G -t freebsd-zfs da2
>> da2a added # gpart add -s 270G -t freebsd-zfs da3 da3a added # gpart
>> add -t freebsd-swap da0 da0b added # gpart add -t freebsd-swap da1
>> da1b added # gpart add -t freebsd-swap da2 da2b added # gpart add -t
>> freebsd-swap da3 da3b added # ***The following lines differ depending
>> on the RAID method # zpool create -o altroot=/mnt -O canmount=off
>> zroot raidz1 da0a da1a da2a da3a # # zpool export zroot # gpart
>> bootcode -p /boot/zfsboot da0 # gpart bootcode -p /boot/zfsboot da1 #
>> gpart bootcode -p /boot/zfsboot da2 # gpart bootcode -p /boot/zfsboot
>> da3 # sysctl kern.geom.debugflags=0x10
>> kern.geom.debugflags: 0 -> 16
>> # dd if=/boot/zfsloader of=/dev/da0a bs=512 oseek=1024
>> conv=notrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.845066 secs (84042 bytes/sec) # dd
>> if=/boot/zfsloader of=/dev/da1a bs=512 oseek=1024 conv=notrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.832603 secs (84411 bytes/sec) # dd
>> if=/boot/zfsloader of=/dev/da2a bs=512 oseek=1024 conv=notrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.815799 secs (84915 bytes/sec) # dd
>> if=/boot/zfsloader of=/dev/da3a bs=512 oseek=1024 conv=notrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.828102 secs (84546 bytes/sec) # zpool
>> import -o altroot=/mnt zroot #
>> # zfs set checksum=fletcher4                                   zroot
>> # zfs set atime=off                                            zroot
>> #
>> # zfs create -o mountpoint=/                                   zroot/ROOT
>>
>> #
>> # zpool set bootfs=zroot/ROOT zroot
>> #
>> # cat << EOF > /tmp/bsdinstall_etc/fstab
>>> # Device        Mountpoint      FStype  Options Dump    Pass#
>>> /dev/da0b       none            swap    sw      0       0
>>> /dev/da1b       none            swap    sw      0       0
>>> /dev/da2b       none            swap    sw      0       0
>>> /dev/da3b       none            swap    sw      0       0
>>> ^D#
>> #
>> # exit
>>
>>
>> # echo 'zfs_enable="YES"' >> /etc/rc.conf # # cat /boot/loader.conf
>> zfs_load="YES"
>> ***Except for 9.3-RELEASE, since the line of "zfs_load="YES"" is
>> originally in ***/boot/loader.conf, addition is unnecessary #
>> # zpool set cachefile=/boot/zfs/zpool.cache   zroot
>> # exit
>>
>> ***In 11.1-RELEASE, the last "zpool set
>> cachefile=/boot/zfs/zpool.cache zroot" failed,
>>      shown as "internal error: failed to initialize ZFS library"
>> ===
>>
>>
>> NOT boot log:
>> ===
>> {1} ok boot disk0
>>
>> SC Alert: Host System has Reset
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>> screen not found.
>> keyboard not found.
>> Keyboard not present.  Using ttya for input and output.
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>>
>>
>> Sun Fire V245, No Keyboard
>> Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
>> OpenBoot 4.22.22, 16384 MB memory installed, Serial #XXXXXXXX.
>> Ethernet address 0:14:4f:XX:XX:XX, Host ID: XXXXXXXX.
>>
>>
>>
>> Initializing     1MB of memory at addr        137ff0a000
>> Initializing     1MB of memory at addr        137ff06000
>> Initializing     1MB of memory at addr        137ff00000
>> Initializing     1MB of memory at addr        137fee0000
>> Initializing     1MB of memory at addr        137fe90000
>> Initializing     1MB of memory at addr        137fdea000
>> Initializing    14MB of memory at addr        137f000000
>> Initializing  2032MB of memory at addr        1300000000
>> Initializing  2048MB of memory at addr        1200000000
>> Initializing  2048MB of memory at addr        1100000000
>> Initializing  2048MB of memory at addr        1000000000
>> Initializing  2048MB of memory at addr         300000000
>> Initializing  2048MB of memory at addr         200000000
>> Initializing  2048MB of memory at addr         100000000
>> Initializing  2048MB of memory at addr                 0
>> Rebooting with command: boot disk0
>> Boot device: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 0,0
>> File and args:
>>    
>>>> FreeBSD/sparc64 ZFS boot block
>>      Boot path:   /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 0,0:a
>> Consoles: Open Firmware console
>> ZFS: i/o error - all block copies unavailable
>> ZFS: can't read MOS object directory
>> ZFS: can't find root filesystem
>>
>> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0 (Fri Jul 21
>> 08:44:42 UTC 2017 root at releng2.nyi.freebsd.org) bootpath=""
>>
>> can't load 'kernel'
>>
>> Type '?' for a list of commands, 'help' for more detailed help.
>> OK reboot
>> Rebooting...
>> Program terminated
>> {1} ok
>> ===
>>
>>
>> VERY SLOW boot log:
>> ===
>> {1} ok boot disk3
>>
>> SC Alert: Host System has Reset
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>> screen not found.
>> keyboard not found.
>> Keyboard not present.  Using ttya for input and output.
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>>
>>
>> Sun Fire V245, No Keyboard
>> Copyright 2006 Sun Microsystems, Inc.  All rights reserved.
>> OpenBoot 4.22.22, 16384 MB memory installed, Serial #XXXXXXXX.
>> Ethernet address 0:14:4f:XX:XX:XX, Host ID: XXXXXXXX.
>>
>>
>>
>> Initializing     1MB of memory at addr        137ff0a000
>> Initializing     1MB of memory at addr        137ff06000
>> Initializing     1MB of memory at addr        137ff00000
>> Initializing     1MB of memory at addr        137fee0000
>> Initializing     1MB of memory at addr        137fe90000
>> Initializing     1MB of memory at addr        137fdea000
>> Initializing    14MB of memory at addr        137f000000
>> Initializing  2032MB of memory at addr        1300000000
>> Initializing  2048MB of memory at addr        1200000000
>> Initializing  2048MB of memory at addr        1100000000
>> Initializing  2048MB of memory at addr        1000000000
>> Initializing  2048MB of memory at addr         300000000
>> Initializing  2048MB of memory at addr         200000000
>> Initializing  2048MB of memory at addr         100000000
>> Initializing  2048MB of memory at addr                 0
>> Rebooting with command: boot disk3
>> Boot device: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 3,0
>> File and args:
>>    
>>>> FreeBSD/sparc64 ZFS boot block
>>      Boot path:   /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 3,0:a
>> Consoles: Open Firmware console
>>
>>
>> ***Slow from here(\|/-\|/- animation)***
>> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0 (Fri Jul 21
>> 08:44:42 UTC 2017 root at releng2.nyi.freebsd.org)
>> bootpath="zfs:zroot/ROOT:"
>> Loading /boot/defaults/loader.conf
>> /boot/kernel/kernel data=0xc47740+0xf4078
>> syms=[0x8+0xd5a38+0x8+0xc746a] /boot/entropy size=0x1000
>> /boot/kernel/zfs.ko text=0x25cc60 data=0xb9d0+0x90050
>> |syms=[0x8+0x1bd38/-+0x8+0x16db2]
>> loading required module 'opensolaris'
>> /boot/kernel/opensolaris.ko text=0x3730 data=0x2c8+0x1a30
>> syms=[0x8+0xe10+0x8+0x968] ***It is slow until the above is
>> displayed.***
>>
>>
>> Hit [Enter] to boot immediately, or any other key for command prompt.
>> Booting [/boot/kernel/kernel] in 9 seconds... Booting
>> [/boot/kernel/kernel] in 8 seconds... Booting [/boot/kernel/kernel] in
>> 7 seconds... Booting [/boot/kernel/kernel] in 6 seconds... Booting
>> [/boot/kernel/kernel] in 5 seconds... Booting [/boot/kernel/kernel] in
>> 4 seconds... Booting [/boot/kernel/kernel] in 3 seconds... Booting
>> [/boot/kernel/kernel] in 2 seconds... Booting [/boot/kernel/kernel] in
>> 1 second... Booting [/boot/kernel/kernel]...
>> -\|/-\|/-\|/-\|/-\|jumping to kernel entry at 0xc00b0000.
>> ===
>>
>> ---
>> Yoshihiko Iwama
>>
>>
>> _______________________________________________
>> freebsd-sparc64 at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
>> To unsubscribe, send any mail to "freebsd-sparc64-unsubscribe at freebsd.org"
>>
> 
> _______________________________________________
> freebsd-sparc64 at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
> To unsubscribe, send any mail to "freebsd-sparc64-unsubscribe at freebsd.org"
> 
> _______________________________________________
> freebsd-sparc64 at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
> To unsubscribe, send any mail to "freebsd-sparc64-unsubscribe at freebsd.org"
> 



More information about the freebsd-sparc64 mailing list