Passing a limited amount of disk devices to jails

[Willem Jan Withagen]

On 12-6-2017 11:48, Willem Jan Withagen wrote:
> On 11-6-2017 02:41, Allan Jude wrote:
>> On 06/10/2017 20:13, Willem Jan Withagen wrote:
>>> On 9-6-2017 16:20, Miroslav Lachman wrote:
>>>> Willem Jan Withagen wrote on 2017/06/09 15:48:
>>>>> On 9-6-2017 11:23, Steven Hartland wrote:
>>>>>> You could do effectively this by using dedicated zfs filesystems per
>>>>>> jail
>>>>>
>>>>> Hi Steven,
>>>>>
>>>>> That is how I'm going to do it, when nothing else works.
>>>>> But then I don't get to test the part of building the ceph-cluster from
>>>>> raw disk...
>>>>>
>>>>> I was more thinking along the lines of tinkering with the devd.conf or
>>>>> something. And would appreciate opinions on how to (not) do it.
>>>>
>>>> I totally skipped devd.conf in my mind in previous reply. So maybe you
>>>> can really use devd.conf to allow access to /dev/adaX devices or you can
>>>> use ZFS zvol if you have big pool and need some smaller devices to test
>>>> with.
>>>
>>> I want the jail to look as much as a normal system would, and then run
>>> ceph-tools on them. And they would like to see /dev/{disk}....
>>>
>>> Now I have found /sbin/devfs which allows to add/remove devices to an
>>> already existing devfs-mount.
>>>
>>> So I can 'rule add type disk unhide' and see the disks.
>>> Gpart can then list partitions.
>>> But any of the other commands is met with an unwilling system:
>>>
>>> root at ceph-1:/ # gpart delete -i 1 ada0
>>> gpart: No such file or directory
>>>
>>> So there is still some protection in place in the jail....
>>>
>>> However dd-ing to the device does overwrite some stuff.
>>> Since after the 'dd if=/dev/zero of=/dev/ada0' gpart reports a corrupt
>>> gpartition.
>>>
>>> But I don't see any sysctl options to toggle that on or off
> 
>> To use GEOM tools like gpart, I think you'll need to unhide
>> /dev/geom.ctl in the jail
>>
>>
> 
> Right, thanx, could very well be the case.
> I'll try and post back here.
> 
> But I'll take a different approach and just enable all devices in /dev
> Since I'm not really needing security, but only need separate compute
> spaces. And jails have the advantage over bhyve that it is easy to
> modify files in the subdomains.
> Restricting afterwards might be an easier job.
> 
> I'm also having trouble expanding /etc/{,defaults/}devfs.rules and have
> 	'mount -t devfs -oruleset'
> pick up the changes.
> Even adding any extra ruleset to the /etc/defaults/devfs.rules does not
> get picked up, hence my toying with /sbin/devfs.

Right,
That will help.

Next challenge is to allow zfs to create a filesystem on a partition.

root at ceph-1:/ # gpart destroy -F ada8
ada8 destroyed
root at ceph-1:/ # gpart create -s GPT ada8
ada8 created
root at ceph-1:/ # gpart add -t freebsd-zfs -a 1M -l osd-disk-1 /dev/ada8
ada8p1 added
root at ceph-1:/ # zpool create -f osd.1 /dev/ada8p1
cannot create 'osd.1': permission denied
root at ceph-1:/ #

--WjW