Announcing: nuOS 0.0.9.1b1 - a whole NEW FreeBSD distro, NOT a fork

[Teske, Devin]

On Jul 8, 2013, at 11:43 AM, Chad J. Milios wrote:

> On 07/08/13 00:12, Teske, Devin wrote:
>> On Jul 7, 2013, at 2:58 PM, Chad J. Milios wrote:
>> [snip]
>> 
>>>            /etc is now a ZFS dataset of its own
>>>                How did we do it?
>>>                    Decades of conventional wisdom says /etc must be on /.
>>>                    Check it out, discuss the whys and the trade-offs.
>> Well, I see in nu_install on GitHub how you're doing it:
>> 
>> You added:
>> 
>> 	init_script="/boot/init.sh"
>> 
>> to /boot/loader.conf, wich -- among other things -- does these two interesting things (variable names changed to make things more clear):
>> 
>> zfs rollback -r $zfs/swap/host at blank
>> NOTE: $zfs is equal to $( /bin/kenv vfs.root.mountfrom ) minus the leading "zfs:"
>> 
>> and
>> 
>> zfs mount $zpool/etc
>> NOTE: $zpool is equal to $zfs from above, leading up to (but not including) the first slash (/).
>> 
>> Cute. Have to say I wasn't aware of the init_script feature of loader.conf(5). Not bad.
> 
> We also had to put one file into the etc directory on the / "beneath" the /etc mount so that /sbin/init can read it before /etc is mounted. There were two or three ways we could do that and each has a tradeoff.
> 

I've been bitten by that.

Getting access to that file that's "beneath" once you've booted the system can be ... less than easy.

I'm interested in your cost/benefit points of having /etc a separate filesystem.

On the face of it, I want to say that "/etc" is (or at least contains) the "core identity" of the machine (and to a lesser extent -- because this is BSD after-all -- /usr/local/etc). In my mind, /etc and /usr/local/etc *are* the machine (metaphorically speaking), so the merits of having it as a separate filesystem are weighed against your desired topology.

If you want to bunch of machines to look and/or act differently, then a shared /etc is precisely what you want. However, without allowing minor changes (ala ZFS clone/snapshot or by way of UnionFS), you'll quickly find that the only way to cope is with role-based scripting in /etc/rc.conf (it is after-all a shell script) or complicated abstraction layers (for example, using netgraph eiface devices with the jail-name inside them so that rc.conf have have jail-specific ifconfig_* lines). But I digress.

I think the better solution to your loading of files "beneath" the eventual /etc filesystem is to throw away the ZFS snapshot/clone method and instead move to a UnionFS approach for /etc.

If you use UnionFS for your /etc, then what you do is for each of the machines that you want *that* /etc to appear, you do something like:

	(as root) mount_unionfs -o below /etc /other/etc

Now /other/etc (assuming it was empty before) looks exactly like /etc.

Pros: With "rm -f <file>; rm -W <file>" (in /other/etc) you can reclaim a file from the underlying /etc. ZFS does not allow you to revert a single file (you can revert the entire volume or filesystem, but not a single file).

Cons: The advantage of having /etc as a ZFS filesystem is probably going to be the compressratio. Using something like lzjb compression on your /etc directory is beneficial (not as beneficial has say /var/log, but by means of having mostly text files, /etc should compress nicely). But... if you *really* need to compress your /etc (that is to say, you're hard-up enough for space that you need the little-savings that you'll gain from compressing /etc), then you're also hard-up enough that you should just set compression on the entire filesystem (nullifying your need to make /etc a separate filesystem -- /etc would get the compression feature from the underlying root filesystem; whatever that is -- zfs filesystem, zpool, zvol, etc.). So again, UnionFS looks like a win unless you *really* want to set separate filesystem features for /etc that you don't set elsewhere.

Were you perhaps after a zfs-/etc for some other reason? because there are other reasons that I'm not getting into. For example, using sysutils/zxfer to make backups of the /etc directory of an entire cloud of machines to a single host. If you don't have /etc as a separate filesystem (and all you want is /etc) then a ZFS stream is of course out of the question and you'll have to resort to rsync. I personally think zxfer is more efficient than rsync but I haven't done the calculations yet to prove it (but it feels like it -- incremental snapshot transfers are pretty darned quick).


> What we did (mv /etc/login.conf.db /boot/etc; ln -s ../boot/etc/login.conf.db /etc/login.conf.db) has the undesirable effect that one must remember to (or be reminded/automated) run cap_mkdb anytime /etc is rolled to a different snapshot or a backup is restored to it (if that changes login.conf).
> 

*nods*


> With our customers at ccsys.com we have a proprietary management thing in userland (and you could lose out on that important event hook if you used anything other than our interface for zfs rollbacks and restoring backups, which we forbid).

Interesting.



> Since our goals at nuos.org are different, i'd like to implement that trigger somewhere better, ideally as-needed and immediate as possible.
> 
> if anyone with more intimate knowledge of how and exactly when login.conf.db gets accessed has any thoughts... It could be a disaster for an admin to think their /etc is in a certain state and have that one file be out of sync. If better minds could chip in, I'm wondering if we're better off editing /sbin/init to run init_script _before_ loading the daemon class from login.conf.db (or explain why thats a bad idea) or if i should just add some sort of hook to run cap_mkdb right when needed using a DTrace script or auditd?
> 

That's an interesting aspect of the boot process I hadn't noticed before (having not used init_script before). I would think that this should be filed as a PR. Seems to me that the init_script should fire first -- but (and this is a guess) it may need to bootstrap the user that the init_script runs as (presumably needing to load the daemon class for said user). While there may be good reason, it certainly violates a principle (that one might be astonished to learn that init_script is not run in a fashion that only the dependencies thereof are required).



> Does anyone think this issue is moot? (Can't we just document this particular specific "gotcha" instance? I don't think so, I abhor any "gotcha" that deviates from behavior people expect from "upstream" fbsd.) Does anyone agree it's important we come as close to perfect a solution as we can?

Thanks for bringing up the issue with init_script. We should look to fix it to make its use capable of handling the use-case you identified (using it to bootstrap a separate /etc).


> Is a separate /etc even worth it to people?

Depends. Everybody? certainly not. Some? Sure. See above example-cases.


> Should i scrap that feature because of this issue?

It sounds like you contorted yourself working around a deficiency in it (a POLA violation in that it has unforeseen dependencies). At the very least, I would think that init could have a fall-back if the file can't be loaded.

Are you putting anything beside the default daemon-class definition in your login.conf "beneath" your true /etc?


> I think we can tighten this up so theres no twisted ankles and no one falling in this rare case but certainly potential manhole. (the manhole i'm talking about is login.conf and login.conf.db being out of sync because the later is a symlink to /boot/etc and someone might rollback to a more restrictive login.conf and think they're covered without running cap_mkdb again but their login.conf.db is actually out of sync and less restrictive in a way that burns them)
> 

Sorry you had to work around that -- you should have filed a PR.



> Devin, thank you IMMENSELY for bsdinstall and especially bsdconfig. I use them both at work and they make life so much better. And thank you for the simplification using kenv. I was unaware of it

On a side-note, I didn't write bsdinstall -- I'm going to maintain it, but I wrote bsdconfig ^_^ (smiles)

Thank you very much for your appreciation. Certainly a labor of love and I'm happy that others have kicked the wheels at least.
-- 
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.