Root on ZFS RAIDZ1
iwama at t3.rim.or.jp
iwama at t3.rim.or.jp
Sat May 5 17:54:14 UTC 2018
Hi Kurt,
I understood your configuration. It is very reasonable. I am satisfied with
the mirror configuration as well.
I tried installing again with RAIDZ1 in the procedure you sent to me.
Unfortunately, the result was the same as before. That is, as follows:
da0 da1 da2 da3
11.1-RELEASE NG NG SLOW SLOW
NG: not boot, SLOW: boot at very slow
On other, eg amd64 platform, RAIDZ1 will work fine, so is there a problem
with the sparc64 boot code?
---
Yoshihiko Iwama
-----Original Message-----
From: Kurt Lidl <lidl at pix.net>
Sent: Saturday, May 5, 2018 6:18 AM
To: iwama at t3.rim.or.jp; freebsd-sparc64 at freebsd.org
Subject: Re: Root on ZFS RAIDZ1
I have my V240 configured to have one set of mirrored drives on the first two disks, and a second set of mirrored disks on the last two disks.
This allows me to have two completely independent OS installs on the machine, and easily switch between them, depending on which disk I boot the system from. I use different ZFS pool names on the different mirrors, so I can, if needed, import the ZFS pool from the other set of disks. Handy for fixing up certain classes of experimental errors on the machine.
For a while, I ran with one set of mirrored disks, and just had non-mirrored zpools set up on the other two disks, one mounted as /usr/src and the other mounted as /usr/obj. That experiment didn't show any huge benefit, so I just went to my current setup, with two sets of independent mirrored disks.
I never tried a RAIDZ of all four disks on the sparc64.
-Kurt
On 5/4/18 5:09 PM, iwama at t3.rim.or.jp wrote:
> Hi Kurt,
>
> Thank you for your reply.
>
> Thank you for providing your achievements and procedures. I will
> try it in your procedure.
>
> Please let me ask a question.
> In your procedure, the RAID method is mirror, but is there a record
> of RAIDZ of 4 disks with V240?
>
> Regards,
> ---
> Yoshihiko Iwama
>
> -----Original Message-----
> From: owner-freebsd-sparc64 at freebsd.org
> <owner-freebsd-sparc64 at freebsd.org>
> On Behalf Of Kurt Lidl
> Sent: Friday, May 4, 2018 10:31 PM
> To: freebsd-sparc64 at freebsd.org
> Subject: Re: Root on ZFS RAIDZ1
>
> I have been using various sparc64 machines with a ZFS only
> installation for quite a while, and found that it works reliably for
> me. My earlier machines were Netra-T1 (all retired now), and
> currently I have both a V240 and three
> V120 machines using this configuration.
>
> Here are my notes that I use during the partitioning phase of the install.
> (ie, start a normal install as discussed in the web page you mention,
> and during the 'Shell' method of partitioning, I run these commands):
>
> You may or may not want to execute the commented out sysctl, which
> will force the ZFS installation to use 4096 byte blocks.
> Given that all the disks I've ever used on sparc64 are really 512byte
> block devices, I'm not sure if this is a great idea or not. I
> typically do run with 4096 byte blocks.
>
> -----------------------------------------------------------------
>
> kldload zfs
>
> # force 4K minimum zfs block size
> #sysctl vfs.zfs.min_auto_ashift=12
>
> # appropriate for a 36GB disk w/4+GB of swap zfssize=30000M #
> appropriate for a 72GB disk w/4+GB of swap zfssize=65500M #
> appropriate for a 72GB disk w/8+GB of swap zfssize=61500M
>
> disks="da0 da1"
> for d in $disks
> do
> gpart destroy -F $d
> gpart create -s VTOC8 $d
> gpart add -t freebsd-zfs -s $zfssize $d
> gpart add -t freebsd-swap $d
> gpart bootcode -p /boot/zfsboot $d
> dd if=/boot/zfsloader of=/dev/${d}a bs=512 oseek=1024 \
> conv=notrunc,sync
> done
>
> # note, we use -R /mnt to set a different root mountpoint P=zroot
> mnt=/mnt zpool create -f -o cachefile=none -O checksum=fletcher4 \
> -R $mnt -m none $P mirror /dev/da0a /dev/da1a
>
> zfs create $P/ROOT
> zfs create $P/ROOT/default
> zpool set bootfs=$P/ROOT/default $P
>
> # because of "-R /mnt" earlier, this is "/mnt" effectively for this
> session zfs set mountpoint=/ $P/ROOT/default
>
> zfs create $P/vartmp
> zfs create $P/usrports
> zfs create $P/usrsrc
> zfs create $P/usrobj
> zfs create $P/home
> zfs set mountpoint=/var/tmp $P/vartmp
> zfs set mountpoint=/home $P/home
> zfs set mountpoint=/usr/ports $P/usrports zfs set mountpoint=/usr/src
> $P/usrsrc zfs set mountpoint=/usr/obj $P/usrobj zfs mount -a
>
> gmirror label -b prefer gswap0 da0b da1b
>
> eeprom boot-device="disk0 disk1"
>
> exit
>
> -----------------------------------------------------------------
>
> At the end of the installation process, you will need to fix up the
> /etc/rc.conf and /boot/loader.conf too:
>
> # Add to /etc/rc.conf:
> clear_tmp_enable="YES"
> ntpdate_enable="YES"
> rwhod_enable="YES"
> rwhod_flags="-m"
> zfs_enable="YES"
>
> # Add to /boot/loader.conf:
> zfs_load="YES"
> geom_mirror_load="YES"
> vfs.zfs.prefetch_disable=1
>
> # Add to /etc/fstab:
> # Device Mountpoint FStype Options Dmp Pass#
> /dev/mirror/gswap none swap sw 0 0
> #md /tmp mfs rw,-s64m 0 0
> tmpfs /tmp tmpfs rw,size=64m,mode=1777 0 0
>
> # turn off ttyv? entries in /etc/ttys
> # turn off ttyu2 entry in /etc/ttys
>
> I hope this helps you get the system working well.
>
> Best wishes,
>
> -Kurt
>
> On 5/4/18 5:08 AM, iwama at t3.rim.or.jp wrote:
>> I want Sun Fire V245 to use FreeBSD-11.1-RELEASE with Root on ZFS
> RAIDZ1.
>> The installation procedure of Root on ZFS is as follow page:
>>
>> https://wiki.freebsd.org/RootOnZFS/VTOC8ZFSBoot/9.x-RELEASE
>>
>> I finished the installation without any problem, but it was some
>> problems at reboot.
>>
>> 1) It do not boot at all
>> 2) boot takes a very long time, more than 3 HOURS
>> 3) It takes some time for boot, about 10 minutes
>>
>> I tried RAIDZ1 with 9.3-RELEASE, RAIDZ2, RAIDZ10 and mirror with
>> 11.1-RELEASE, the results were as follows, da0-3 are the name of boot
>> disk. (Fire V245 has four disks):
>>
>> RAIDZ1(zpool create -o altroot=/mnt -O canmount=off zroot raidz1 da0a
>> da1a da2a da3a)
>> da0 da1 da2 da3
>> 11.1-RELEASE NG NG SLOW SLOW
>> 9.3-RELEASE NG NG SLOW SLOW
>>
>> RAIDZ2(zpool create -o altroot=/mnt -O canmount=off zroot raidz2 da0a
>> da1a da2a da3a)
>> da0 da1 da2 da3
>> 11.1-RELEASE SLOW SLOW SLOW SLOW
>>
>> RAIDZ10(zpool create -o altroot=/mnt -O canmount=off zroot mirror
>> da0a da1a mirror da2a da3a)
>> da0 da1 da2 da3
>> 11.1-RELEASE slow slow slow slow
>>
>> mirror(zpool create -o altroot=/mnt -O canmount=off zroot mirror da0a
>> da1a da2a da3a)
>> da0 da1 da2 da3
>> 11.1-RELEASE OK OK OK OK
>>
>> OK: OK(boot at normal speed), NG: 1), SLOW: 2), slow: 3)
>>
>>
>> From the above results and the logs below, there is no problem
>> with mirror, but it is a big problem in RAIDZ1 or RAIDZ2.
>> Has anyone been running Root on RAIDZ1 on the sparc64 platform?
>>
>>
>> Install steps(only the part entered manually):
>> ===
>> # zpool labelclear -f /dev/da0a
>> # zpool labelclear -f /dev/da1a
>> # zpool labelclear -f /dev/da2a
>> # zpool labelclear -f /dev/da3a
>> #
>> # sysctl kern.disks
>> kern.disks: cd0 da3 da2 da1 da0
>> # gpart destroy -F da0
>> da0 destroyed
>> # gpart destroy -F da1
>> da1 destroyed
>> # gpart destroy -F da2
>> da2 destroyed
>> # gpart destroy -F da3
>> da3 destroyed
>> # gpart create -s VTOC8 da0
>> da0 created
>> # gpart create -s VTOC8 da1
>> da1 created
>> # gpart create -s VTOC8 da2
>> da2 created
>> # gpart create -s VTOC8 da3
>> da3 created
>> #
>> # gpart add -s 270G -t freebsd-zfs da0 da0a added # gpart add -s 270G
>> -t freebsd-zfs da1 da1a added # gpart add -s 270G -t freebsd-zfs da2
>> da2a added # gpart add -s 270G -t freebsd-zfs da3 da3a added # gpart
>> add -t freebsd-swap da0 da0b added # gpart add -t freebsd-swap da1
>> da1b added # gpart add -t freebsd-swap da2 da2b added # gpart add -t
>> freebsd-swap da3 da3b added # ***The following lines differ depending
>> on the RAID method # zpool create -o altroot=/mnt -O canmount=off
>> zroot raidz1 da0a da1a da2a da3a # # zpool export zroot # gpart
>> bootcode -p /boot/zfsboot da0 # gpart bootcode -p /boot/zfsboot da1 #
>> gpart bootcode -p /boot/zfsboot da2 # gpart bootcode -p /boot/zfsboot
>> da3 # sysctl kern.geom.debugflags=0x10
>> kern.geom.debugflags: 0 -> 16
>> # dd if=/boot/zfsloader of=/dev/da0a bs=512 oseek=1024
>> conv=notrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.845066 secs (84042 bytes/sec) # dd
>> if=/boot/zfsloader of=/dev/da1a bs=512 oseek=1024 conv=notrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.832603 secs (84411 bytes/sec) # dd
>> if=/boot/zfsloader of=/dev/da2a bs=512 oseek=1024 conv=notrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.815799 secs (84915 bytes/sec) # dd
>> if=/boot/zfsloader of=/dev/da3a bs=512 oseek=1024 conv=notrunc,sync
>> 466+1 records in
>> 467+0 records out
>> 239104 bytes transferred in 2.828102 secs (84546 bytes/sec) # zpool
>> import -o altroot=/mnt zroot #
>> # zfs set checksum=fletcher4 zroot
>> # zfs set atime=off zroot
>> #
>> # zfs create -o mountpoint=/ zroot/ROOT
>>
>> #
>> # zpool set bootfs=zroot/ROOT zroot
>> #
>> # cat << EOF > /tmp/bsdinstall_etc/fstab
>>> # Device Mountpoint FStype Options Dump Pass#
>>> /dev/da0b none swap sw 0 0
>>> /dev/da1b none swap sw 0 0
>>> /dev/da2b none swap sw 0 0
>>> /dev/da3b none swap sw 0 0
>>> ^D#
>> #
>> # exit
>>
>>
>> # echo 'zfs_enable="YES"' >> /etc/rc.conf # # cat /boot/loader.conf
>> zfs_load="YES"
>> ***Except for 9.3-RELEASE, since the line of "zfs_load="YES"" is
>> originally in ***/boot/loader.conf, addition is unnecessary #
>> # zpool set cachefile=/boot/zfs/zpool.cache zroot
>> # exit
>>
>> ***In 11.1-RELEASE, the last "zpool set
>> cachefile=/boot/zfs/zpool.cache zroot" failed,
>> shown as "internal error: failed to initialize ZFS library"
>> ===
>>
>>
>> NOT boot log:
>> ===
>> {1} ok boot disk0
>>
>> SC Alert: Host System has Reset
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>> screen not found.
>> keyboard not found.
>> Keyboard not present. Using ttya for input and output.
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>>
>>
>> Sun Fire V245, No Keyboard
>> Copyright 2006 Sun Microsystems, Inc. All rights reserved.
>> OpenBoot 4.22.22, 16384 MB memory installed, Serial #XXXXXXXX.
>> Ethernet address 0:14:4f:XX:XX:XX, Host ID: XXXXXXXX.
>>
>>
>>
>> Initializing 1MB of memory at addr 137ff0a000
>> Initializing 1MB of memory at addr 137ff06000
>> Initializing 1MB of memory at addr 137ff00000
>> Initializing 1MB of memory at addr 137fee0000
>> Initializing 1MB of memory at addr 137fe90000
>> Initializing 1MB of memory at addr 137fdea000
>> Initializing 14MB of memory at addr 137f000000
>> Initializing 2032MB of memory at addr 1300000000
>> Initializing 2048MB of memory at addr 1200000000
>> Initializing 2048MB of memory at addr 1100000000
>> Initializing 2048MB of memory at addr 1000000000
>> Initializing 2048MB of memory at addr 300000000
>> Initializing 2048MB of memory at addr 200000000
>> Initializing 2048MB of memory at addr 100000000
>> Initializing 2048MB of memory at addr 0
>> Rebooting with command: boot disk0
>> Boot device: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 0,0
>> File and args:
>>
>>>> FreeBSD/sparc64 ZFS boot block
>> Boot path: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 0,0:a
>> Consoles: Open Firmware console
>> ZFS: i/o error - all block copies unavailable
>> ZFS: can't read MOS object directory
>> ZFS: can't find root filesystem
>>
>> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0 (Fri Jul
>> 21
>> 08:44:42 UTC 2017 root at releng2.nyi.freebsd.org) bootpath=""
>>
>> can't load 'kernel'
>>
>> Type '?' for a list of commands, 'help' for more detailed help.
>> OK reboot
>> Rebooting...
>> Program terminated
>> {1} ok
>> ===
>>
>>
>> VERY SLOW boot log:
>> ===
>> {1} ok boot disk3
>>
>> SC Alert: Host System has Reset
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>> screen not found.
>> keyboard not found.
>> Keyboard not present. Using ttya for input and output.
>> Probing system devices
>> Probing memory
>> Probing I/O buses
>>
>>
>> Sun Fire V245, No Keyboard
>> Copyright 2006 Sun Microsystems, Inc. All rights reserved.
>> OpenBoot 4.22.22, 16384 MB memory installed, Serial #XXXXXXXX.
>> Ethernet address 0:14:4f:XX:XX:XX, Host ID: XXXXXXXX.
>>
>>
>>
>> Initializing 1MB of memory at addr 137ff0a000
>> Initializing 1MB of memory at addr 137ff06000
>> Initializing 1MB of memory at addr 137ff00000
>> Initializing 1MB of memory at addr 137fee0000
>> Initializing 1MB of memory at addr 137fe90000
>> Initializing 1MB of memory at addr 137fdea000
>> Initializing 14MB of memory at addr 137f000000
>> Initializing 2032MB of memory at addr 1300000000
>> Initializing 2048MB of memory at addr 1200000000
>> Initializing 2048MB of memory at addr 1100000000
>> Initializing 2048MB of memory at addr 1000000000
>> Initializing 2048MB of memory at addr 300000000
>> Initializing 2048MB of memory at addr 200000000
>> Initializing 2048MB of memory at addr 100000000
>> Initializing 2048MB of memory at addr 0
>> Rebooting with command: boot disk3
>> Boot device: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 3,0
>> File and args:
>>
>>>> FreeBSD/sparc64 ZFS boot block
>> Boot path: /pci at 1e,600000/pci at 0/pci at a/pci at 0/pci at 8/scsi at 1/disk at 3,0:a
>> Consoles: Open Firmware console
>>
>>
>> ***Slow from here(\|/-\|/- animation)***
>> FreeBSD/sparc64 ZFS enabled bootstrap loader, Revision 1.0 (Fri Jul
>> 21
>> 08:44:42 UTC 2017 root at releng2.nyi.freebsd.org)
>> bootpath="zfs:zroot/ROOT:"
>> Loading /boot/defaults/loader.conf
>> /boot/kernel/kernel data=0xc47740+0xf4078
>> syms=[0x8+0xd5a38+0x8+0xc746a] /boot/entropy size=0x1000
>> /boot/kernel/zfs.ko text=0x25cc60 data=0xb9d0+0x90050
>> |syms=[0x8+0x1bd38/-+0x8+0x16db2]
>> loading required module 'opensolaris'
>> /boot/kernel/opensolaris.ko text=0x3730 data=0x2c8+0x1a30
>> syms=[0x8+0xe10+0x8+0x968] ***It is slow until the above is
>> displayed.***
>>
>>
>> Hit [Enter] to boot immediately, or any other key for command prompt.
>> Booting [/boot/kernel/kernel] in 9 seconds... Booting
>> [/boot/kernel/kernel] in 8 seconds... Booting [/boot/kernel/kernel]
>> in
>> 7 seconds... Booting [/boot/kernel/kernel] in 6 seconds... Booting
>> [/boot/kernel/kernel] in 5 seconds... Booting [/boot/kernel/kernel]
>> in
>> 4 seconds... Booting [/boot/kernel/kernel] in 3 seconds... Booting
>> [/boot/kernel/kernel] in 2 seconds... Booting [/boot/kernel/kernel]
>> in
>> 1 second... Booting [/boot/kernel/kernel]...
>> -\|/-\|/-\|/-\|/-\|jumping to kernel entry at 0xc00b0000.
>> ===
>>
>> ---
>> Yoshihiko Iwama
>>
>>
>> _______________________________________________
>> freebsd-sparc64 at freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
>> To unsubscribe, send any mail to "freebsd-sparc64-unsubscribe at freebsd.org"
>>
>
> _______________________________________________
> freebsd-sparc64 at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
> To unsubscribe, send any mail to "freebsd-sparc64-unsubscribe at freebsd.org"
>
> _______________________________________________
> freebsd-sparc64 at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-sparc64
> To unsubscribe, send any mail to "freebsd-sparc64-unsubscribe at freebsd.org"
>
More information about the freebsd-sparc64
mailing list