glusterfs + FreeBSD [was Re: CEPH + FreeBSD]

[Jordan Hubbard]

[ Adding Justin Clift to CC line ]

> On Sep 6, 2015, at 3:18 PM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> 
> Hmm. From a quick look at their web page (I looked once before as well), I don't
> think RiakCS has what I need to do pNFS in a reasonable (for me) amount of effort.
> Two things that glusterFS has that I am hoping to use (and I don't think RiakCS has
> either of these) are:
> - A Fuse file system interface which allows the kernel nfsd to access the store as
>  a file system, so that it can provide the metadata services (NFS without the reads/writes).
> - A userland NFSv3 server in each node which will allow the node to act as a data server.
> 
> If I am wrong and RiakCS does support a VFS file system interface (via Fuse or ???), then
> please correct me.

You are not wrong.  RiakCS is basically just a distributed object store with an Amazon S3 compatible API.  It doesn’t attempt to expose the object store as a filesystem or provide any other modes of access to it other than via S3 or its own “database API” since it’s also (at its heart) a distributed database.

In all fairness to Basho, I also think that was probably the right call.  It’s not a file system and doesn’t pretend to be one, it’s just a distributed database that can store arbitrary numbers (and sizes) of objects in a cluster with fault-tolerance, multi-data center tenancy and so on (I also have no connection with Basho so if it sounds like I’m trying to sales pitch it or something, I’m not, I’m simply trying to be accurate in describing it given that we’re also evaluating it as a possible technology to incorporate into FreeNAS, for which the topic of S3-compatible object storage comes up fairly frequently.

Now glusterfs is another kettle of fish, and like I said, I think the bigger question there is just what FreeBSD’s relationship with it is.  The https://wiki.freebsd.org/GlusterFS wiki was created over a year ago and seems to be suffering from an overall lack of specific goals.  Does the FreeBSD project want glusterfs and/or is there anyone in the FreeBSD community even interested in deploying it in a production scenario?  Is anyone willing to actually maintain a port, just as the minimum price of admission?

The bug tracking this, namely https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194409, has been open for almost a year now and while it looks like ‘craig001” has been trying fairly hard to get 3.7.2 up and working, he’s hitting some walls and it’s not even clear he’s in contact with the right folks.

As the previous glusterfs mail thread indicated, I was also happy to add an earlier version of his port to the FreeNAS repo and play with it there, though I was forced to take it back out of FreeNAS when we ran into a number of issues just trying to make it work with the Linux glusterfs reference VMs we set up.  Should the underlying port get some love again, it would be the simplest thing in the world for me to refresh our copy of it and uncomment the line which adds glusterfs to the FreeNAS manifest (and being a storage appliance, it’s rather of an obvious test bed for this since people running FreeNAS are also the target audience for technologies like this).

If getting glusterfs up as a stable, working substrate is also a prerequisite for getting pNFS, then I think a lot of us in the storage industry would be the first to say “What do we have to do?  How can we help?”   I already volunteered to send Rick an AMD64 architecture machine for development, but I suspect that’s actually the least of our obstacles right now. :-)

- Jordan