CEPH + FreeBSD

[Willem Jan Withagen]

On 6-12-2015 23:52, Willem Jan Withagen wrote:
> On 6-12-2015 21:56, Jordan Hubbard wrote:
>>
>>> On Dec 4, 2015, at 3:12 AM, Willem Jan Withagen <wjw at digiware.nl>
>>> wrote:
>>>
>>> Talking to Sage Weill, he said that one of the main things to keep 
>>> FreeBSD-Ceph up and running, is the possibility to actually run
>>> the automated builds and tests on a FreeBSD system. So that is
>>> something I/we need to think about.
>>
>> I think sufficient cloud resources to run a jenkins CI environment or
>> something which does the builds on commit triggers and runs
>> regression tests are the easy part - we run dozens (if not hundreds)
>> of virtual FreeBSD instances in multiple datacenters to facilitate
>> our own testing and donating a VM to this effort would be trivial.
> 
> I have for this more or less the same feeling. Given the number of
> companies actually running FreeBSD in all shapes and form, would it not
> be very hard to find either real hardware, let alone getting VMs to run
> tests....

Hi Jordan,

In the NFS Plan B thread I promised you a sort of idea how to help.
Below the answer I wrote to an individual FreeBSD user that asked the
same. I still haven't quite figured out what the way forward is.
And as you state it, it also depends on what you'd like to do with Ceph.
Which can be different for each and every one of us.
Below is the answer I gave to the individual.

I also talked to Ed Schouten, but he's more of the "fix the application"
direction. And my work is more like:
	this is the hamer
	lest make a FreeBSD nail.
Ed wants to make screws, because that is the better solution.

So feel free to pick an choose, and or give direction.

Biggest thing to get working for me ATM is ceph-disk, because that will
get things to start installing. Making it packageble, and something for
other to start playing with. And it is only going to work out if people
start using it.

--WjW

====
My corrent work is all in
	https://github.com/wjwithagen/ceph
and the branch I think has the most complete work is:
	 wip-wjw-freebsd-cmake

But due to my inexperience with GIT, I'm always a bit carefull to
commit, because I need to be able to find them again and move over to
another branch to crete a pull for it. But perhaps that is also
something you can help with: I'm relatively new to git, but slowly
getting there.

I could certainly send you a diff for everything that is outstanding.

Things on my agenda:
Currently I'm fighting getting some of the tests in the testframework
working. Python and bash. There are a few more python problems to be solved.

The next major step will be that we get the ceph-disk script working,
because that is needed for ceph-deploy. And by then we are able to run
installs.

But there are also some more complex problems to resolve.

The is a challenge in the signaling code:
 - at_exit routines do not (always) get called.
 - some of the daemons need to be signaled more than once before they
   die.
I'd expect these 2 to be related.

Our AIO lib is not compatible with Linux's so that will need a wrapper,
or even more complex code. This will be required to start using
bluestoreFS, the simple FS for Ceph.

I'd also would like to start using ZFS a FileStore. There is code for
btrfs, so there would be some examples to borough from.

There is KeyUtils on Linux, which is used as secret keystore. We need to
port it, or rip it out by supplying an empty wrapper.

The penultimate being a kernel device that actually created RBD-devices
as a "native" device for FreeBSD to use. And I was thinking about using
gated-stuff for that.