ZFS and Jail :: nullfs mount :: nothing visible from host

[Miroslav Lachman]

SK wrote on 2016/12/09 12:36:
> Thanks Miroslav, I get the picture now. Please see my reply inline

>> I mean your command "zfs list", because normally "zfs list" inside
>> jail print: "no datasets available" :)
>>
> OK, considering that I have the setup as I explained before, and have
> run zfs jail testJail gT/JailS/testJail, I can see the complete dataset
> along with the ones that are NOT part of the jail. So, whatever dataset
> the host can see, I can see from inside the jail. However, I cannot do
> anything with the dataset from inside the jail.

I am not sure, maybe it is not possible to hide them when you need to 
manage zfs inside jail.
If you can live with not managing zfs inside but from the host, then you 
can use enforce_statfs=2. Then you will see just a root dataset inside jail.

enforce_statfs=0 ~ you will see all datasets and partitions from the host

enforce_statfs=1 ~ you will see all related to this jail (parents, devfs 
etc)

enforce_statfs=2 ~ only root mount is visible

>>> But, what I would really like to have
>>>
>>> a) ONLY the relevant datasets for a jail are visible and can be
>>> manipulated from within the jail. I do not mind if they are visible from
>>> host (in fact, I might prefer that -- not manipulate, just see and maybe
>>> take snapshot of what is there -- helps in centralizing backups). But
>>> the Jails /must not/ see each others' datasets
>>
>>
>> zfs create gT/JailS/testJail
>> zfs set jailed=on gT/JailS/testJail   << Did you set this property?
> Now this is an interesting bit. I tried this, and as soon as I ran the
> command, the dataset vanished :P
>
> Not only that, I could not run jail any more. Given that gT/JailS is
> mounted on /JailS and the path parameter in jail.conf is
> /JailS/testJail, I am not surprised that the jail did not run (it
> initially complained about not being able to mount /dev, as it cannot
> find /JailS/testJail/dev)
>
> As a workaround, I removed mount.devfs, mount.procfs (that complained
> too), mount.fdesc (complained too), and then the jail ran
>
> But now that I do not have devfs, I could not do anything with zfs -- I
> could not even see them. So, manipulation from within the jail or
> outside the jail was no longer possible.

Interesting. All documentation says jailed=on must be set.

>> # (populate & start jail)
>>
>> zfs jail testJail gT/JailS/testJail
>>
>>> b) if that is not achievable, maybe not allow the jails to see the
>>> complete dataset hierarchy -- just make them feel that they are where
>>> they are in a root, but still be able to create datasets that would
>>> magically show up in the respective jails. This way, the total control
>>> is from the host itself, where no one has access to, but the datasets
>>> are restricted to different jails.
>>
>> What is visible is controlled by enforce_statfs values. If you create
>> /tank/jail/alpha and set this path to you first jail no other jail
>> will know about it.
> This I believe is where I am stuck at the moment. How do you set this
> path to the jail? Apparently running zfs jail testJail gT/JailS/testJail
> did not stop the testJail from seeing gT/Data or gT/JailS/Moving -- in
> fact, they became visible after that script was run.
>
> Any suggestion/pointers is greatly welcome.
>
> Out of a little bit of frustration (since I was unable to find any
> proper documentation on jail.conf -- there is nothing under
> /etc/default, there is nothing on the man page -- I could not even
> figure out how to define a zfs as the root/fs for the jail!), I have
> started looking into ezjail now -- given that everyone seem to claim it
> can do what I had been unable to do through command line. If my sense
> and intelligence is well enough, I might be able to find out how it is
> done.

"Everybody" say "use ezjail" because it was the first tool to manipulate 
jails available for the masses. I tried it after I learned all things 
about jails the hard way and then I realised ezjail is doing strange 
things in some cases. I know it evolved, but I you need to use some tool 
there are some better tools (in my opinion) which were developed with 
ZFS features from the start.
You can try iocage or cbsd. They also can manage bhyve guests.

Miroslav Lachman