Tips for shell based partitioning during install

Jim Ohlstein jim at
Thu May 4 23:03:12 UTC 2017


On 05/04/2017 06:34 PM, Peter Pauly wrote:
>>> That sounds like extremely huge amount of total swap space.
> At this point it's just a test system (proof of concept), but I welcome all
> criticism. I'm using sixteen 100GB drives in VMware.

Ok, I wasn't going to say anything, but since you have an open mind, a 
16 drive RAIDZ2 is asking for failure, especially if the drives are all 
the same make and age. A second (or third) disk failure is a high risk 
during resilver with this many drives.

If it were me, I'd install the OS on a two drive zpool. Create swap only 
on these disks (4-8GB total should be enough).

Then you have 14 drives to configure for data storage. I might go with 
two 7 drive RAIDZ3's or two 6 drive RAIDZ2's with a hot spare for each.

> On Thu, May 4, 2017 at 6:00 PM, Valeri Galtsev <galtsev at>
> wrote:
>> On Thu, May 4, 2017 4:48 pm, Peter Pauly wrote:
>>> I got a lot farther, but my system won't boot. I get this message:
>>> ZFS: i/o error - all block copies unavailable
>>> ZFS: can't read MOS of pool zroot
>>> gptzfsboot: failed to mount default pool zroot
>>> I'm pasting my build instructions below starting with the part where I
>>> begin partitioning. Maybe you'll see some obvious thing that I did wrong
>>> (thanks):
>>> 8. Under Partitioning, choose Shell
>>> 9. Create GPT Disks:
>>> gpart create -s gpt da0
>>> gpart create -s gpt da1
>>> gpart create -s gpt da2
>>> gpart create -s gpt da3
>>> gpart create -s gpt da4
>>> gpart create -s gpt da5
>>> gpart create -s gpt da6
>>> gpart create -s gpt da7
>>> gpart create -s gpt da8
>>> gpart create -s gpt da9
>>> gpart create -s gpt da10
>>> gpart create -s gpt da11
>>> gpart create -s gpt da12
>>> gpart create -s gpt da13
>>> gpart create -s gpt da14
>>> gpart create -s gpt da15
>>> 10. Add the boot partition to each drive:
>>> gpart add -s 512k -t freebsd-boot da0
>>> gpart add -s 512k -t freebsd-boot da1
>>> gpart add -s 512k -t freebsd-boot da2
>>> gpart add -s 512k -t freebsd-boot da3
>>> gpart add -s 512k -t freebsd-boot da4
>>> gpart add -s 512k -t freebsd-boot da5
>>> gpart add -s 512k -t freebsd-boot da6
>>> gpart add -s 512k -t freebsd-boot da7
>>> gpart add -s 512k -t freebsd-boot da8
>>> gpart add -s 512k -t freebsd-boot da9
>>> gpart add -s 512k -t freebsd-boot da10
>>> gpart add -s 512k -t freebsd-boot da11
>>> gpart add -s 512k -t freebsd-boot da12
>>> gpart add -s 512k -t freebsd-boot da13
>>> gpart add -s 512k -t freebsd-boot da14
>>> gpart add -s 512k -t freebsd-boot da15
>>> 11. Add the swap partition to each drive:
>>> gpart add -s 8G -t freebsd-swap -l swap0 da0
>>> gpart add -s 8G -t freebsd-swap -l swap1 da1
>>> gpart add -s 8G -t freebsd-swap -l swap2 da2
>>> gpart add -s 8G -t freebsd-swap -l swap3 da3
>>> gpart add -s 8G -t freebsd-swap -l swap4 da4
>>> gpart add -s 8G -t freebsd-swap -l swap5 da5
>>> gpart add -s 8G -t freebsd-swap -l swap6 da6
>>> gpart add -s 8G -t freebsd-swap -l swap7 da7
>>> gpart add -s 8G -t freebsd-swap -l swap8 da8
>>> gpart add -s 8G -t freebsd-swap -l swap9 da9
>>> gpart add -s 8G -t freebsd-swap -l swap10 da10
>>> gpart add -s 8G -t freebsd-swap -l swap11 da11
>>> gpart add -s 8G -t freebsd-swap -l swap12 da12
>>> gpart add -s 8G -t freebsd-swap -l swap13 da13
>>> gpart add -s 8G -t freebsd-swap -l swap14 da14
>>> gpart add -s 8G -t freebsd-swap -l swap15 da15
>> That sounds like extremely huge amount of total swap space. Imagine, the
>> machine starts swapping in and out of swap these amounts of data. It will
>> put the machine on its knees, it will be totally unreponsive, I figure. Do
>> I miss something? Am I wrong about something?
>> Valeri
>>> 12. Add the main partition to the remaining space on each drive:
>>> gpart add -t freebsd-zfs -l disk0 da0
>>> gpart add -t freebsd-zfs -l disk1 da1
>>> gpart add -t freebsd-zfs -l disk2 da2
>>> gpart add -t freebsd-zfs -l disk3 da3
>>> gpart add -t freebsd-zfs -l disk4 da4
>>> gpart add -t freebsd-zfs -l disk5 da5
>>> gpart add -t freebsd-zfs -l disk6 da6
>>> gpart add -t freebsd-zfs -l disk7 da7
>>> gpart add -t freebsd-zfs -l disk8 da8
>>> gpart add -t freebsd-zfs -l disk9 da9
>>> gpart add -t freebsd-zfs -l disk10 da10
>>> gpart add -t freebsd-zfs -l disk11 da11
>>> gpart add -t freebsd-zfs -l disk12 da12
>>> gpart add -t freebsd-zfs -l disk13 da13
>>> gpart add -t freebsd-zfs -l disk14 da14
>>> gpart add -t freebsd-zfs -l disk15 da15
>>> 13. Install the Protective MBR (pmbr) and gptzfsboot loader to all
>> drives:
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da1
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da2
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da3
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da4
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da5
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da6
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da7
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da8
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da9
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da10
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da11
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da12
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da13
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da14
>>> gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da15
>>> 14. Load ZFS kernel module:
>>> kldload /boot/kernel/opensolaris.ko
>>> kldload /boot/kernel/zfs.ko
>>> 15. Create the zfs pool:
>>> zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f
>> zroot
>>> raidz2 /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk2 /dev/gpt/disk3
>>> /dev/gpt/disk4 /dev/gpt/disk5 /dev/gpt/disk6 /dev/gpt/disk7
>> /dev/gpt/disk8
>>> /dev/gpt/disk9 /dev/gpt/disk10 /dev/gpt/disk11 /dev/gpt/disk12
>>> /dev/gpt/disk13 /dev/gpt/disk14 /dev/gpt/disk15
>>> 16. Create the zfs datasets:
>>> zfs create -o mountpoint=none zroot/ROOT
>>> zfs create -o mountpoint=/ zroot/ROOT/default
>>> zfs create -o mountpoint=/tmp -o exec=on -o setuid=off zroot/tmp
>>> zfs create -o mountpoint=/usr -o canmount=off zroot/usr
>>> zfs create zroot/usr/home
>>> zfs create -o setuid=off zroot/usr/ports
>>> zfs create zroot/usr/src
>>> zfs create -o mountpoint=/var -o canmount=off zroot/var
>>> zfs create -o exec=off -o setuid=off zroot/var/audit
>>> zfs create -o exec=off -o setuid=off zroot/var/crash
>>> zfs create -o exec=off -o setuid=off zroot/var/log
>>> zfs create -o atime=on zroot/var/mail
>>> zfs create -o setuid=off zroot/var/tmp
>>> 17. Set the mount point of the root for newly created datasets
>>> zfs set mountpoint=/zroot zroot
>>> 18. Set correct permissions on the temp directories:
>>> chmod 1777 /mnt/tmp
>>> chmod 1777 /mnt/var/tmp
>>> 19. Tell zfs where to find the boot file system:
>>> zpool set bootfs=zroot/ROOT/default zroot
>>> 20. Create a directory for the zpool cache and tell zfs where to find it:
>>> mkdir -p /mnt/boot/zfs
>>> zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot
>>> 21. Set the canmount=noauto so that default boot environment (BE) does
>> not
>>> get mounted if a different boot environment is chosen from the boot menu:
>>> zfs set canmount=noauto zroot/ROOT/default
>>> 22. Add command which will be picked up later to build rc.conf: and
>>> loader.conf
>>> echo ‘zfs_enable=\”YES\”’ >> /tmp/bsdinstall_etc/rc.conf.zfs
>>> echo ‘kern.geom.label.disk_ident.enable=\”0\”’ >>
>>> /tmp/bsdinstall_boot/loader.conf.zfs
>>> echo ‘kern.geom.label.gptid.enable=\”0\”’ >>
>>> /tmp/bsdinstall_boot/loader.conf.zfs
>>> 23. exit
>>> 24. At this point, the base system will be installed.
>>> 25. Enter the root password.
>>> 26. Under Network Configuration, select  vmx0
>>> 27. Would you like to configure IPv4 for this interface:  Yes
>>> 28. Would you like to use DHCP to configure this interface:  No
>>> 29. Enter the following information:  IP Address:, Subnet Mask:
>>>, Default Router:
>>> 30. Would you like to configure IPv6 for this interface:  No
>>> 31. Search:  <blank>, IPv4 DNS #1:, IPv4 DNS #2:
>>> 32. On the Time Zone Selector screen, choose America -- North and South,
>>> then scroll down and choose New York.
>>> Choose Eastern Time and when asked, does EST look reasonable, choose Yes
>>> 33. Choose Skip then Skip again to skip the time adjustment.
>>> 34. On the services you would like to have started at boot screen,
>>> choose:,
>>> sshd, ntpd and dumpdev
>>> 35. On the system hardening page, ENABLE ALL OPTIONS.
>>> 36. Would you like to add users to the installed system now:  No
>>> 37. On the final configuration screen, choose Exit
>>> 38. Before exiting, would you like to open a shell?  Yes
>>> 39. zpool set cachefile= zroot
>>> exit
>>> 40. Installation is complete screen:  Reboot
>>> On Wed, May 3, 2017 at 4:46 PM, Ben Woods <woodsb02 at> wrote:
>>>> On Wed, 3 May 2017 at 2:31 am, Peter Pauly <ppauly at> wrote:
>>>>> I'm using the option in the installer where you go out to a shell
>>>>> prompt
>>>>> during the partitioning step in the installer on FreeBSD 11-Release and
>>>>> booted off of the CD. All is going well until I get to this step:
>>>>> zpool create zroot raidz2 /dev/gpt/disk0 /dev/gpt/disk1 /dev/gpt/disk3
>>>>> ...
>>>>> etc.
>>>>> cannot mount '/zroot': failed to create mountpoint
>>>>> The instructions when going out to the shell say I have to mount the
>>>>> filesystem under /mnt but /mnt is read-only.
>>>>> I tried to use the Guided Auto (ZFS) but it doesn't work with more than
>>>>> 10
>>>>> drives.
>>>>> What am I doing wrong?
>>>> Hi Peter,
>>>> When I do manual zfs partitioning during installs, i follow the commands
>>>> used by the actual bsdinstall scripts (the ones that would have been
>>>> executed if I used the auto mode).
>>>> A copy of them can be viewed online here:
>>>> scripts/zfsboot?view=markup#l1313
>>>> Essentially the zpool create command needs to have:
>>>> zpool create -o altroot=/mnt -O compress=lz4 -O atime=off -m none -f
>>>> zroot
>>>> raidz2 /dev/gpt/disk0 ...
>>>> Regards,
>>>> Ben
>>>>> --
>>>> --
>>>> From: Benjamin Woods
>>>> woodsb02 at
>>> _______________________________________________
>>> freebsd-questions at mailing list
>>> To unsubscribe, send any mail to
>>> "freebsd-questions-unsubscribe at"
>> ++++++++++++++++++++++++++++++++++++++++
>> Valeri Galtsev
>> Sr System Administrator
>> Department of Astronomy and Astrophysics
>> Kavli Institute for Cosmological Physics
>> University of Chicago
>> Phone: 773-702-4247
>> ++++++++++++++++++++++++++++++++++++++++
> _______________________________________________
> freebsd-questions at mailing list
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at"

Jim Ohlstein

More information about the freebsd-questions mailing list