Setting up ZFS - Filesystem Properties and Installing on Root

[APseudoUtopia]

On Mon, Nov 21, 2011 at 10:14 AM, Daniel Staal <DStaal at usa.net> wrote:
>
> On Sun, November 20, 2011 10:34 pm, APseudoUtopia wrote:
>> Hello,
>>
>> I'll be setting up a server with ZFS on 9.0-RELEASE (when it's
>> released...). I've never used ZFS before, and although I've been
>> reading quite a bit about it, I have some questions.
>>
>> My plan is to use RAID-Z1 across 4 disks. I'll be using GPT, and I
>> would like the root to be ZFS as well. I found a guide:
>> http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1
>>
>> In step #4, it has you create boot, swap, and zfs partitions on all 3
>> (which would be 4 in my case) disks. Then, in step #5, you install the
>> bootloader into all 3 (4) drives. Why do you need boot and swap
>> partitions on EACH disk? It seems to me that you would only need disk
>> 1 to have boot, swap, and zfs, and the other 3 disks only have one
>> partition (using the entire drive) for zfs's pool. Does it have to do
>> with the RAIDZ1 setup? Even then, I don't understand it because it's
>> not disk mirroring, it's RAID. The BIOS is set to look on one specific
>> disk for the loader, not all of them. It seems I'm not understanding
>> something entirely here.
>
> As someone else said, the BIOS will search attached drives.  And remember
> that one of the points to using a ZFS raid is that you can get hotswap
> drives.  If your one drive with a bootloader fails, you can hotswap it and
> not notice that it was that drive, which then means your next boot will
> fail.
>
> Personally, I dedicated one small drive to the boot code, and put
> everything else in ZFS.  There are varying opinions on whether that is a
> good idea.  ;)
>
> Swap in theory can be put on the ZFS volume - and I have done so - but I
> would not recommend it.  It's caused lockups in some situations, which
> would have been avoided by putting swap on it's own volume.
>
>> Also, with ZFS, you can have an unlimited number of filesystems,
>> correct? I've been trying to figure out the best way to create these
>> filesystems with the appropriate flags (specifically: atime,
>> compression, devices, exec, quota, readonly, and setuid). If, for
>> example, I set devices=off and suid=off on the tank/var filesystem, it
>> is applied to the children filesystem, such as, /var/log, /var/db, and
>> so on? The flags/properties can be changed on-the-fly, correct? If,
>> for example, I set a filesystem noexec, but later realize I need exec,
>> I can change it without issue?
>
> Yep.  As long as you didn't set the volume that has the zfs tools to
> noexec.  ;)
>
>> Does anyone with zfs experience have any tips on creating a filesystem
>> layout, in terms of which filesystems to create and what
>> flags/properties? Would it be bad to set noatime, nosuid, nodev, and
>> noexec all on the tank, then allow each property appropriately for
>> each directory as necessary? As in, set the whole tank noexec, but
>> allow exec for /bin, /usr/home, /usr/local/bin, etc.?
>
> Well, you already linked to a decent guide to the filesystem layout.
> Following it won't get you too far wrong.  A couple of thoughts though:
>
> One thing that might have been useful in my experience is not having / be
> /zroot.  If I'd set /zroot/root to / (and built everything else below
> that) I'd be able to play around a bit more, by setting up other
> filesystems under /zroot. (Including things like making a new root volume,
> or excluding volumes from snapshots.)  Note that would cause changes in
> the bootloader code from the wiki example.
>
> And don't be to worried about getting it 'right the first time': Because
> ZFS allows you to create filesystems so easily, you can play around with
> it.  It's not too hard to move a directory tree to a new filesystem, even
> on a live box.  ;)  You can create copies, try things out, and then use
> whatever's the best system for you.
>
> As for noexec on /...  I'd be hesitant.  Without being an absolute master
> on what is run from where under all circumstances in FreeBSD, it would
> worry me.  (Does /etc need exec?   How about /root?  /boot?  etc.)  I'd
> rather err on the side of turning it off where I know I don't need it,
> instead of turning it on where I do.  If I forget one, it's less likely to
> break my system.  ;)
>
> The one other place where I'd disagree with the wiki page you linked would
> be on compression: If you have a fairly recent processor in the box, I'd
> recommend gzip for just about everywhere.  In most cases it'll actually
> speed up I/O.
>
> Oh, and if you are thinking of using dedup, you'll need sha256 for the
> checksum.  But I think it'll do that automatically if you turn on dedup.
>
> Daniel T. Staal
>
> ---------------------------------------------------------------
> This email copyright the author.  Unless otherwise noted, you
> are expressly allowed to retransmit, quote, or otherwise use
> the contents for non-commercial purposes.  This copyright will
> expire 5 years after the author's death, or in 30 years,
> whichever is longer, unless such a period is in excess of
> local copyright law.
> ---------------------------------------------------------------
>

Ok, thank you for the advice. It's on a quad-core xeon, so, in
reference to your suggestion, I'll turn on compression for the entire
pool. I did read that having swap in zfs may cause problems, so I'll
be sticking to the traditional freebsd-swap partition.

Another quick question about swap: If I have 4 drives, with 512MB
swap, the system uses all 4 swap partitions, correct? So it's not like
it'd be going to waste? I'd have a total of 2 GB swap?

Thanks!